electronics

Feather HUZZAH Temperature Monitor

I recently visited the cabin, and it was cold. Like, excessively cold. Like, 37 degrees, which is perilously close to pipes-freezing cold. The thermostat shouldn’t allow that to happen. I hadn’t been there for more than a month, so I don’t know how long there had been a problem, but clearly, the thermostat or furnace wasn’t working. I went into the crawlspace under the house, pulled some panels off the furnace, and did a bit of troubleshooting on my own. Then I did a bit of troubleshooting with an HVAC guy on the phone. Eventually, we determined that something was, indeed, broken. The HVAC guy came out, replaced the controller board on the furnace, and I had heat again.

At the office, I build and maintain complicated software systems. Any sufficiently complicated system is going to have unpredictable failure modes. I accept that I can’t avoid all possible failure modes, but once I recognize a critical failure class, I build monitors to alert me to any failure in that class. It’s what I do in software, so it makes sense to do it in hardware as well. I don’t know all the failure modes of the heating system in the cabin, but failure of the heating system is certainly a failure class that could have very bad (as in, expensive) consequences.

I recently became aware of Adafruit’s new Arduino-compatible line of development boards, Feather. The Feather HUZZAH, is particularly interesting, as it has built-in WiFi (based on the ESP8266 chipset), and costs only $16. With a Feather HUZZAH and a temperature sensor, like the MPC9808 I2C breakout, I could put together an inexpensive monitor. I happened to have a spare, small I2C OLED display that I could add to the mix for a bit of feedback.

Components

The code to initialize and control the temperature sensor and OLED is short and easy. The loop() portion of the sketch reads the temperature, puts it on the display, and if 15 minutes have passed since the last time data was sent to the server, send the temperature to the server and reset timer variable. Finally, shut down the temp sensor and sleep for two seconds. It looks like this:

void loop() {
  float f = tempsensor.readTempF();

  display.clearDisplay();
  display.setCursor(0,0);
  display.print(f, 1);
  display.print('F');
  display.display();

  if (millis() - send_timer >= 1000 * 60 * 15) {
    WiFiClient client;
    if (!client.connect(host, httpPort)) {
      Serial.println("connection failed");
    }
    else {
      client.print(String("GET ") + url +
        "?code=" + mac + "&tval=" + f + " HTTP/1.1\r\n" +
        "Host: " + host + "\r\n" + 
        "Connection: close\r\n\r\n");
      send_timer = millis();
    }
  }

  tempsensor.shutdown_wake(1);
  delay(2000);
  tempsensor.shutdown_wake(0);
}

The only problem I had was that when I tried uploading the sketch to the HUZZAH, I got the error,

warning: espcomm_sync failed
error: espcomm_open failed

A bit of research indicated that to upload a sketch, I’d need to connect Pin 0 to ground and reset the unit (either by power cycling it, or by hitting the reset button).

Pin 0 to Ground

With Pin 0 held to ground, the sketch uploaded. After connecting the temp sensor and OLED, the device seemed to measure the temperature accurately. I took some dimension measurements, and designed an enclosure in TinkerCAD. By the time I had soldered the connections, the two pieces of the enclosure had finished printing.

Enclosure

The last component for this project is a server-side piece that could record the temperature. In the simplest case, I could set up a page that listens for incoming data, and sends me an email or text message when a temperature is posted below some threshold. But I wanted also to be able to see trends over time. So I needed to store readings in a database. Since I might want to have multiple temperature monitors running in several locations, I need to record a source with each temperature reading. To normalize the database, I split the source and measurement into two tables, like this:

mysql> describe sources;
+-------+-------------+------+-----+---------+----------------+
| Field | Type        | Null | Key | Default | Extra          |
+-------+-------------+------+-----+---------+----------------+
| id    | int(11)     | NO   | PRI | NULL    | auto_increment |
| code  | varchar(64) | YES  | MUL | NULL    |                |
| name  | varchar(64) | YES  |     | NULL    |                |
+-------+-------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)
mysql> describe temps;
+-------------+--------------+------+-----+-------------------+-----------------------------+
| Field       | Type         | Null | Key | Default           | Extra                       |
+-------------+--------------+------+-----+-------------------+-----------------------------+
| id          | int(11)      | NO   | PRI | NULL              | auto_increment              |
| source_id   | int(11)      | NO   | MUL | NULL              |                             |
| measured_at | timestamp    | NO   |     | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| temperature | decimal(4,1) | NO   |     | NULL              |                             |
+-------------+--------------+------+-----+-------------------+-----------------------------+
4 rows in set (0.00 sec)

After I had recorded temperature measurements for several days, I had enough data to start putting something on a graph. Rather than building a graphing mechanism from scratch, I repurposed some D3 code that I had written for my UltraSignup Visualizer (which was, at least in part, repurposed from my MMT Graph project). The D3 code pulls data (as JSON) from a PHP script that retrieves temperature measurements and timestamps from some specified source. It then draws the graph, and adds the (slightly smoothed) measurements.

// Smooth temperature readings over avglen measurements
var temperatures = [];
var cur_temp = 0;
var avglen = 8;
var i = 0;
// Seed the running average
while (i < (avglen - 1) && i < results.length) {
  cur_temp += (1.0 * results[i].t);
  i++;
}
// Populate the running average (cur_temp) as a FIFO list of avglen length
while (i < results.length) {
  cur_temp += (1.0 * results[i].t);
  temperatures[temperatures.length] = {x: results[i].d, y: (cur_temp / avglen)};
  i++;
  cur_temp -= (1.0 * results[i - avglen].t);
}

// Create the SVG line function
var line = d3.svg.line()
   .interpolate("basis")
   .x(function(d, i) { return xScale(new Date(d.x)); })
   .y(function(d) { return yScale(d.y); });

// Add the data to the graph using the line function defined above
svg.append("path")
   .attr("d", line(temperatures))
   .attr('class', 'rank_line')
   .style("fill", "transparent")
   .style("stroke", "rgba(71, 153, 31,.8)")
   .style("stroke-width", 1.25);

Temperature Graph

Now that everything works, I’d like to make a few more of these devices. The main costs are $15.95 for the Feather HUZZAH, $4.95 for the temperature sensor, $17.50 for the OLED display, a few dollars for a micro-USB cable and power supply, and some cents for a few inches of wire and a few grams of PLA (for the 3D printed enclosure). For the cost of the device, the display is disproportionately expensive. Once the device is running, the purpose is to record temperature remotely. If I replace the OLED with a single NeoPixel (which’ll run about $1) that flashes some color code to indicate status, I don’t get the onboard temperature readout, but I DO get the entire device for around $23 (plus the micro-USB cable and power supply). So the next iteration will replace the OLED with a NeoPixel. Stay tuned.

electronics

Jeep Computer Update

A year and a half ago, I wrote about a spanky little computer I built for my Jeep. It had been working like a boss for a little more than a year before it started to malfunction. Whereas it used to acquire a GPS fix almost instantly, it started to take several minutes to find the satellites. And the “trip timer” function (which measured trip time and average speed) no longer worked correctly. Every time the car was powered on, the clock was reset to 7:59pm or 6:59pm (depending on daylight savings status). After getting a GPS fix, the time would correct itself, but as far as the computer was concerned, the trip started at the pre-fix time, often adding several hours to the actual trip time.

My first suspicion was that the external antenna had gone bad. It’s encased in black plastic, and it sits on top of the dashboard. I imagined that the hot, summer sun baked it somehow, so the computer could only use its internal antenna, tucked (with the rest of the main processing unit) tightly under the dashboard. I ordered a new antenna, and started designing a small, protective case that I could print in white PLA, so the antenna would have some protection from direct sunlight. I decided that it would also make sense to replace the black enclosure that I had printed for the display with a white enclosure, and put a back on it for additional protection from the sun. When I pulled the display out of the car, I realized that the heat had taken a toll. The black enclosure had melted, and was drooping around the display.

Enclosure Before

Before: The enclosure was nice and square when it was first installed.

Enclosure After

After: A year and a half in the sun took its toll.

I literally had to cut away the enclosure to get the display out of it, as the PLA had so tightly molded itself around the back circuit board. Fortunately, I now have my own 3D printer, so I could use the same STL file to print a new copy of the enclosure, and design a cap to provide some shade for the back side of the display circuit board.

Finally, the main processing unit — the Arduino with the GPS shield, and all the assorted connections — was shoehorned into an awkwardly shaped, generic enclosure that I had purchased at Radio Shack [moment of silence, please]. So that was due for a new, custom enclosure. I made a pretty simple box, with some cutouts for the power and USB connectors, and holes where I could mount the GPS antenna connector and temperature sensor plug.

New Enclosure

After putting all of that together, the new antenna I had ordered arrived. I tried it out… And it made no difference. So on to the next step of troubleshooting. I remembered that the GPS shield had a built-in real time clock (RTC). That would have been the module responsible for maintaining the time when the device was powered off. I assumed also that keeping accurate time between power cycles would allow the GPS to establish a faster fix on the satellites. I pulled out the RTC battery (which was a CR1220 form factor). My multimeter happens to be dead now (due to a blown fuse) so I couldn’t test it. Fortunately, batteries are cheap. I got a new battery, and now the time is correct upon powering up, and the unit gets a fix within a few seconds. In the interim, I did make a minor tweak to the Arduino code. Previously, I depended on the unit maintaining the correct time between power cycles, so I started the trip calculation (time and average speed) immediately. Now, I wait until the GPS has a fix to start the calculations. In the normal case, the fix only takes a few seconds. However, the next time the battery dies, it will still affect the time it takes to find the satellites, but it will no longer cause the trip calculations to be totally erroneous.

 

programming

Sudoku, part 1

I know, I know. I’ve been MIA from this blog. And I feel just terrible about it. I haven’t abandoned it. I always have plenty of projects going on. Lately, I’ve just been wrapped up with a large project that has taken quite a bit of time. Lots of 3D printing and photography and working with new and interesting bits of software. I’m not ready to say much else about it right now.

Lately, though, I’ve had a bit of time to work on other projects. I decided to dig up a half-done Sudoku related project, to give it some polish. The crux of the project is a Sudoku solver (that I intend to write about at some point in the future). I decided that I needed an interface to the solver, so I put together this handy little web interface.

It allows the user to enter an initial puzzle, or to pull a random, pre-made puzzle. After the initial board is set, the user can allow for forbid impossible moves (that is, moves that would conflict with a number that is already set in the same row, column, or block). The user can also show pencil marks. Pencil marks are indications of playable numbers at each position.

Pencil Marks

I haven’t yet connected to solver to the interface. That’ll come later. The first priority was getting the board to respond to input correctly. Most of the work for managing the available moves on the board is done by my JavaScript class, Board. The Board class takes care of making sure that a move can be made, and updating all affected squares. The function to place a move in a square will only place the move if it is a legal move, or if the initial board has been set and the option to permit impossible moves is allowed. If the move can be placed, all related squares (squares in the same row, column, or block) are updated.

this.place = function(val, r, c) {
  var av = this.available[r][c];
  var placed = false;
  if ((this.setDone && getEl('impossibles_permit').checked) ||
      1 == av[val]) {
    this.placed[r][c] = val;
    this.updateAffectedLocations(r, c);
    placed = true;
  }
  return placed;
}

Higher level functions that interact with the front end operate outside of the class. The callback used when a move is played either tries to place the move if the keypress is between 1 and 9 (ASCII 49-57), or clear the square on backspace or delete (ASCII 8 or 127, respectively). If the move cannot be placed, blink the square that is blocking it.

function setSquare(e) {
  var k = (e == null ? event.keyCode : e.which);
  if (activesq) {
    var r = activesq.getAttribute('row');
    var c = activesq.getAttribute('col');
    var activepen = getPen(activesq);
    var activepencil = getPencil(activesq);

    if (48 < k && k < 58) {
      var val = String.fromCharCode(k);
      if (myBoard.place(val, r, c)) {
        activepencil.style.visibility = 'hidden';
        activepen.innerHTML = val;
      }
      else {
        var sq = myBoard.findBlocker(val, r, c);
        blinkSquare(sq, errcolor);
      }
    }
    else if (8 == k || 127 == k) {
      myBoard.unplace(r, c);
      activepen.innerHTML = '';
      if (getEl('pencil_show').checked) {
        activepencil.style.visibility = 'visible';
      }
    }
  }
  clearListener();
  return false;
}

A little JavaScript, and a little nice styling… That’s about all that’s involved. I’ll get around to hooking up the solver in the next few weeks.

knitting

Quadruple Cross Mitts

Download the pattern: Quadruple Cross Mitts

I made the first pair of these mitts in 2009. I came up with a rough sketch, then took notes on the design as I went along. I intended to write up the pattern immediately afterward. A year later, when I still hadn’t written up the pattern, I decided I needed to make another pair to make sure that my notes were correct. I did that, and again, failed to formally write up the pattern. My notes sat in a binder for lo these many years, until I finally decided that I would, once again, knit the mitts as a refresher, then write up the pattern. This time it stuck.

Quadruple Cross Mitts

Skills

  • Knit and Purl
  • K2Tog and SSK
  • Circular cast on
  • Circular bind off – Since you’ll be binding off individual fingers a “jog” will be quite noticeable. If you are not confident in this skill, I recommend reviewing the TECHKnitting review of circular bind offs.
  • Cable 1 left and Cable 1 right – If you don’t know how to cable without an extra needle, this would be a good project for learning. There are many tutorials about cabling without an extra needle, such as this or this.
  • M1 with reverse loop
  • Picking up stitches

Joining Fingers

Quad Cross MittsThe trick with gloves is not to leave holes between the fingers. I’ve tried several strategies to avoid the inter-digital void; the strategy described in this pattern is the one that I find works best. It is repeated several times in the pattern, and in fact, is a significant contributor to the complexity of the description. If you get your head around the finger joins before beginning the pattern, you’ll find that the whole thing becomes much less complex.

The primary point to recognize is that the finger join involves turning one “tube” into two. We’ll call them Tube A and Tube B. Upon separating Tube A from Tube B, you’ll continue knitting Tube A, and set aside Tube B for later. And that brings us to the second point to recognize: that Tube A and Tube B will not be symmetrical. You’ll be adding a few extra stitches between the tubes, but those stitches will be added differently to Tube A than to Tube B (and most of those stitches will disappear shortly after the base of the join).

So here we go… Get your knitting visualization caps on. You’re knitting the main tube of the work, and you are ready to start a finger. You have arranged the stitches so that the stitches for the finger are on three needles, and the remainder of the stitches are on waste yarn. On the third needle of the finger, you’ll cast on two new stitches with reverse loops, and join it to the first needle. This is the beginning of Tube A. On the next round, you’ll knit the two new stitches through the back loop (to tighten them up). On the round after that, at the stitch before the two new stitches you’ll SSK then K2Tog, effectively removing the two new stitches.

Tube B will come from the remaining stitches. Put those stitches on three needles. On one of those needles, you’ll pick up four stitches from the base of Tube A. As with Tube A, you’ll knit one round keeping all stitches. Then on the following round, you’ll SSK the first of the new stitches with the stitch before it, and you’ll K2Tog the last of the new stitches with the stitch after it. Now you have the base of Tube B.

Gauge

Hand MeasurementThese mitts are meant to be knit at a gauge that would, for most garments, be wrong for the yarn. I’d recommend starting with a yarn that recommends size 8 needles for 4 or 5 stitches per inch, and knit a swatch on size 6 needles. You should end up with a gauge around 5.5 stitches per inch (or 22 stitches per four inches). With the 40 stitches in the main part of the pattern, this gauge results in a tube of approximately 7.25 inches in circumference. That fits well — gives the right amount of negative ease — on a hand that is approximately 7.5 inches around (measured at the knuckles, around the base of the fingers).

 

Download the pattern: Quadruple Cross Mitts

snippets

Sheetz

insideShe raised her eyebrows toward the middle of her forehead. This, he knew, was to express an emotion that was composed of 40% concern and 60% skepticism. It seemed to him that she was unable to experience simple feelings. Not that she was unable to experience feelings—rather, her feelings always seemed to be some unnamed composite of named emotions. Until he met her, had he thought about it, he would have believed himself to be a sort of emotional genius, being able to distinguish between even subtle variations of happiness and joy, or sadness, sorrow and despair. But she… She lived her life like a master chef seasoning each moment with a unique blend emotional herbs and spices. His emotional palate had expanded in the months they had been together. And in this dish, he could distinguish concern and skepticism.

Had she been with her girlfriends, the look would have been enough. They would have known what she meant. But in her brain, in her overly-developed frontal lobe, a signal emerged to notify other areas of her mind that there was a non-negligible chance that he had missed the entire meaning of the look she threw his way, and this issue was far too important for her to risk failing to convey her passionately held view. Her language center was first to respond, producing the words in just such a tone that he would have been unable to miss their deeper meaning. “You mean, at that gas station where you can get food?”

“No,” he said, perhaps a shade too pointedly. He was aiming at 65% flippant, 25% sincere and 10% firmly resolved. He suddenly feared that he lacked the fine vocal control to accomplish such a technical maneuver, and that he had overshot the “firmly resolved”. He pulled back for the briefest moment to regroup before continuing. “I mean, at that eating establishment where you can refuel your car.”

Mmmm.

electronics

3D Printing and Custom Enclosures

I finally got around to finishing my Jeep computer project. I had gotten the Arduino and display working, and I had wired everything together. However, as of my previous post about the project, neither the display nor the Arduino were in enclosures, and the cabin of the Jeep was festooned with wires.

The first step in cleaning up the mess was to purchase an inexpensive, generic, plastic, rectangular enclosure from Radio Shack. I drilled some holes to mount the connections for the Arduino/GPS unit. I put it together and tucked it behind the dash, nice and neat. So most of the wires were gone. The only remaining messy part was the display. I still had the bare display nestled e’er so gently in a knit cap that I’d leave on top of the dash as a pillow for my electronics. I wanted an enclosure that would fit snugly around the display, and that I could mount on the dashboard.

At some point along the way, I learned that the Washington DC public library system has a 3D printing service. For a minimal cost, they would make a print of an object. Fantastic! I just needed to figure out how to create a model. I learned that I was going to need to use a CAD program to create an .stl (STereoLithography) file, which seems to be one of the primary file formats in the 3D printing world.

Of course, I didn’t want to spend thousands of dollars on a CAD program that would take years to learn. Fortunately, there are free, easy-to-learn options, such as SketchUp or TinkerCAD. SketchUp is a native program that runs on Macs and Windows computers. While I have a couple of Macs, and I run various versions of Windows in virtual machines for testing purposes, at home, my primary computer is Linux. TinkerCAD is a very simple, web-based CAD program that works well in any modern browser.

After a basic exploration of TinkerCAD, I was ready to go about designing the custom enclosure for my display. The display is a 2.8″ TFT LCD Touchscreen Breakout from Adafruit. I spent some time searching for specs that list the dimensions, but no such specs were to be found. So I pulled a tape measure out of my knitting kit, and built the CAD model as I measured the dimensions.

TinkerCAD and the Display

My first attempt included a back plane. Having no experience with 3D printing, I didn’t realize that in order to support the back plane during the printing process, the printer would have to lay down a grillwork of plastic that I would have to remove after the fact.

3D Grillwork

I attempted to remove the grillwork, but the plastic is surprisingly sturdy. (In fact, I was originally worried that the 2mm walls of my model would be flimsy, but it ended up being rock solid… Err, in a plastic sort of way.) So I removed the back in the CAD file, and resubmitted. I got back an enclosure that fits the display perfectly.

Enclosure

Enclosure

Enclosure

Enclosure

I considered adding grooves to the enclosure so I could print a separate back plane that could be attached and detached. In the end, I went with simplicity, and I just used a bit of Gorilla Tape for the back. I’ve mounted it on the dashboard (again, with Gorilla Tape until I settle on a more permanent solution), and it satisfies all of my greatest hopes and desires (with respect to a 3D printed enclosure anyway).

Enclosure

Enclosure

 

electronics

Jeep Seat Heaters

Once you own a car with seat heaters, it’s hard to go back. The old VW had heated seats; the new Jeep did not. Clearly, this state of affairs could not stand.

I found that I could get some nice, neoprene seat covers with built-in seat heaters made by Wet Okole. The Wet Okoles came with heating elements in both the butt-area and the back-area, whereas some aftermarket heaters only heat the butt. I used Quadratec’s “designer” for Wet Okoles. When the seat covers arrived, I installed them, and there was much rejoicing.

Wet Okole Seat Covers

Each seat had a cigarette lighter plug for power, and a push-button switch that allowed setting the heater element to Off, Low, Medium, or High.

Original Switch

Here we get to the problem: While convenient for a quick connection, I didn’t want wires dangling from the seats to the cigarette lighter. Further, I only had one cigarette lighter. Even splitting the circuit for the cigarette lighter wasn’t a great solution, as each seat could potentially draw a little more than 10 amps, and the lighter was on a 20 amp fuse. Splitting the circuit would mean that I’d risk blowing the fuse each time both seats were on full. The Jeep conveniently has a spare 20 amp circuit on the fuse block behind the dash, but it’s an unswitched circuit. If I used that, it’d only be a matter of time before I’d leave the car with a seat heater on, and I’d come back to a dead battery.

After using the seat heaters for a while, I decided that they were keepers, so it was worth investing in a more permanent solution to the power problem — and a solution that would allow both seats to be heated at the same time. I needed to add at least two new 15 or 20 amp, switched circuits to the car, and I wanted controls that were integrated into the dashboard somehow.

For the controls, I started looking around for switches that could put into some existing blanks on the dash. However, after bashing in a heater vent (by transporting some furniture in the passenger seat), I stumbled upon the perfect solution in a Daystar replacement vent with an integrated switch panel.

Vent Switches

The drawback of the rocker switches is that I would no longer have Low or Medium settings. The seat heaters would either be off, or fully on. But really, who need a “lightly warmed” bum in winter? If it’s cold enough to turn ’em on, turn ’em on ALL THE WAY, I say!

For the circuits, I found that Painless Performance makes three- and seven-circuit add-on fuse blocks. I decided to go with the seven-circuit block to give me room for expansion in future, yet-to-be-conceived projects (such as my Arduino-based trip computer). That gave me four new switched circuits, and three new constant circuits, all at 20 amps.

The parts arrived, and so I got to connecting all the pieces. The trickiest decision was where to mount the new fuse block. I had initially intended to mount it behind the glove compartment, next to the existing internal fuse block, but there wasn’t enough room. Perhaps I could have found another spot behind the dash, but I didn’t want to put it somewhere that would require ripping open the dash to access it (in case I should need to replace a fuse). There was an empty spot in the engine compartment (for a second battery, I suppose, to power a winch that I’m unlikely to add) that seemed like a good candidate. Being in the engine compartment, I wanted to add a bit of protection to the block, as it wasn’t marketed as a weatherproof component. So rather than mounting it directly, I mounted it to the inside of a small tupperware bin. I drilled a few ventilation holes in the bin, and a larger hole to run the wires, then mounted the bin in the engine compartment.

Placement of fuse block

The new fuse block is in a ventilated tupperware bin, mounted near the back-driver-side of the engine compartment.

The fuse block had three sets of wires:

  1. Two wires to connect to the positive and negative poles of the battery to power the circuits.
  2. A single wire that needed to be connected to an existing switched circuit. This wire poweres an internal relay that controlled the switching of the four switched circuits in the fuse block.
  3. Seven hot wires for the seven new circuits.

Since the fuse block was already in the engine compartment, running the first set of wires to the battery was fairly trivial. The rest of the wires, though, had to make it into the cabin, which meant getting them through the firewall. I spent more than a few minutes looking for an accessible, existing run through the firewall, and was met with no success. I refered to Dr. Google, and learned that a hard, rubber plug near the gas pedal is the preferred channel — just make a hole straight through it. I made the hole, and pulled the wires through.

Wires run through the firewall

To get the wires through the firewall, I had to make a hole in a rubber plug that seemed to exist for exactly that purpose.

For the relay wire, I tapped into the hot line for the cigarette lighter — I just cut away a centimeter of insulation, joined the relay wire, and wrapped it up neat and tidy with electrical tape. The remainder of the work consisted of running wires up and down behind the dash: hot circuit wires to switches, switches to positive wires for the seats, switches to ground, seats to ground.

As of this blog post, the seats have been keeping my bum warm for more than two winters. A boy could hardly ask for more.

Switches

programming

UltraSignup Visualizer


Instructions

  • In the text box at the top of the graph, enter the full name of a runner whose results can be found on UltraSignup, then hit enter.
  • The points on the graph represent individual race results for the given runner. Move your mouse over a point to see details of that race.
  • The line represents the evolution of the runner’s UltraSignup rank.
  • Timed events (eg, 12-hour races, 24-hour races) appear as empty circles. It seems that as of mid-October, 2014, timed events are included in the ranking. However, it is not clear to me if that change is retroactive, and in some circumstances, I cannot get my calculation of the ranking to line up with their calculation of the ranking. So if you have a large number of timed events in your history, the line I’ve calculated might be e’er so slightly off. The ranking reported below the graph is the official number, provided by UltraSignup.

Background

[Update: The friendly folks at UltraSignup came across this, and they liked it. I worked with them to get it integrated into the official runner results page. So now you can click the “History” link just below a runner’s overall score on UltraSignup and see the plot on the results page. Though if you like the spanky transitions between runners, you still need to come here.]

In the world of ultrarunning, it seems that the ranking calculated by UltraSignup has become the de facto standard for ranking runners. I think that part of the reason for its acceptance is its simplicity. A runner’s rank in a single race is just the ratio of the winner’s finish time to the runner’s finish time. So if you win a race, you get a 100%; if you take twice as long as the winner, you get a 50%. The overall ranking is a single number that represents an average of all of a given runner’s race rankings. If you were to look up my results on UltraSignup, you would see that as of this moment of this blog post, my 10+ years of racing ultras has been boiled down to a ranking of 88.43% over 48 races.

Of course, with simplicity comes inflexibility. What that number doesn’t capture is change over time. By summing up my results as a single number, it’s hard to see how my last few years of Lyme-impaired running have affected my rank, or how my (hoped-for) return to form will affect it. I was curious to see how runners progress over time, and how it affects the UltraSignup rank. In looking at the details of how UltraSignup delivers their rank pages, I noticed that the results come as JSON strings. Therefore, I realized, I wouldn’t even have to do any parsing of irregular data. I could just pull the JSON, and use my handy D3 skillz to put the results in a scatter plot.

I won’t go into great depth about implementation details. If you happen to be interested, you can go to the source. A passing familiarity with D3 would be helpful, but familiarity with only vanilla Javascript should allow you to get the gist.

Oh, and be aware that since this pulls data from UltraSignup, it’s entirely possible that it will stop working someday, either because they change the way they deliver data, or because they don’t like third parties creating mashups with their data. Also, this doesn’t work on Internet Explorer 8, or earlier. Sorry ’bout that!

programming

Revoice

Some years ago, I was in need of a new and unique way to taunt a friend. He had made a habit of being especially obnoxious in an certain online forum. But it’s not his obnoxiousness that really bothered me. In fact, many friends and acquaintances would consider me to be chief among purveyors of obnoxiousness.  No… It’s not the obnoxiousness that bothered me. It was the repetition within the obnoxiousness that brought me to action. Every day, it was the same rants, the same complaints, the same stock phrases.

I had, for some time, tried to taunt him by twisting his words, offering clever (according to me) puns, countering him with facts and offering the straight-up observation that he was saying the same thing, day after day. It was like throwing spitballs at a steamroller. Clearly, I needed to step up my game.

My first thought was to compile his most-used stock phrases, create bingo cards with them and distribute those cards to other participants of the forum. That way, even if we had to put up with this repetitive obnoxiousness, at least we could derive some fun from it—and maybe someone could win a prize!

As much fun as that might have been, I decided that I wanted something unique, and bingo had been done. (It hadn’t been done in this particular case. But I wasn’t the first one to think of creating a bingo game based on the phrases someone frequently says.) So I came up with the idea of writing a program that would generate new posts for the forum. The posts would be in the style of the individual of my choosing—using the same vocabulary and phrasing—but would essentially be nonsense. This idea had that dual benefits of being an effective method of taunting as well as being an interesting project.

I had forgotten about this little project for many years. Recently, though, I came across an old, broken laptop. I knew that this laptop had some old projects on it (including a simple 3D drawing program I wrote in college, and some signal processing and computer learning code I had written for a long-extinct dot-com that rode that early-2000s bubble with the best of ’em). I decided to pull the data off the hard drive (a process that ended up being quite a project in itself). I thought my taunting program might be on that disk. But it was nowhere to be found. After some more thought about where and when I wrote the program, I realized that I had written it on a computer from a later era that had since experienced a catastrophic disk failure.

Rather than being disappointed that I had lost that little gem, I decided to recreate it. I recalled that I had written it as a short PERL script that only took a couple hours to code. Although I haven’t touched PERL in six or seven years (other than to make trivial tweaks to existing code), I remembered the premise on which I based the original program, and PERL is certainly the best readily-available tool for this job.

To understand how the program works, you need to understand that everyone has characteristic patterns in their writing or speech—a sort of linguistic fingerprint, or voiceprint. You can do all sorts of analysis on an individual’s language to determine various aspects of that voiceprint. The trick is to find some set of aspects that are not only descriptive, but that could be used generatively.

One component of that voiceprint is vocabulary—the set of words that someone knows and uses. So I could take some sample text, extract the vocabulary used in that text, and use those words in random order. Unfortunately, that would end up as a jumbled mess. The first problem is that most of us share the majority of our vocabulary. There are a few hundred words that make up that majority of what we say. It’s only at the periphery of our vocabulary—words that hold some very specific meaning and that are used infrequently, like “periphery”—where we start to notice differences.

To accomplish my goal, I was going to need a system that would not only use the same words that are in a sample text, but would also use them in a similar, but novel, way. I could take the idea from above—to use the vocabulary—and add the notion of frequency. That is, I wouldn’t just randomly pick words from a vocabulary list. Instead, I would note the frequency of each word used, and pick words with the same probability. For example, if the word “the” makes up 3% of the sample text, then it would have a 3% likelihood of being picked at any particular step in the generated text.

But we still have the problem that the resulting text wouldn’t have any coherence beyond the word level. It would just be a jumble of words strung together. To add coherence, we need context. Context means that we need to look at the way that words are strung together. We can do that by looking at tuples—ordered sets—of words. For the sake of settling on a reasonable number, let’s pick 3. Let’s look at 3-tuples. This paragraph starts with the 3-tuple {But,we,still}. The next 3-tuple is {we,still,have}. Then {still,have,the}, then {have,the,problem}, and on and on.

Looking at tuples, we get a small amount of context.  My question when I started this project was whether that context was enough to build a generative system that could speak in the voice of some training text. Since the tuples are sequences that appear with some known frequency, and since one tuple could be chained to the next by using those known frequencies, I had high hopes.

To understand how this would work, we could train it on a piece of my own writing: my Hellgate 100k race report from 2007. Say I was in the middle of generating new text based on that post, using 3-tuples. Now, say that last two words in my generated text are “going to”. (Forget about how I reached that point, just assume that I’m there.) I need to pick the next word. Based on earlier analysis, I know that the following table shows all of the 3-tuples taken from that post, which start with {going,to}. The number next to each 3-tuple is the number of times the 3-tuple appears in the post.

{going,to,run} (1)
{going,to,win} (1)
{going,to,be} (1)

To be consistent with the sample text, the next word needs to be “run”, “win” or “be”. Based on the frequencies, there is an equal chance of choosing any of the options. Now say that of those options, with those frequencies, we happen to choose “be” as the next word. Our generated text is “going to be”. So we start over, looking for the tuples that start with {to,be}.

{to,be,able} (2)
{to,be,here} (1)
{to,be,very} (1)
{to,be,a} (1)
{to,be,of} (1)
{to,be,running} (1)
{to,be,jogging} (1)
{to,be,at} (1)
{to,be,gathered} (1)

Let’s pick “running”, making our generated text “going to be running”. And on to the next step.

{be,running,for} (1)
{be,running,.} (1)
{be,running,at} (1)

We pick “at”, resulting in “going to be running at”. We are starting to get some text that appears to be somewhat familiar (if you’ve read the sample text) and is syntactically correct, but might be entirely new. (Notice that one of the options in the above example is punctuation. By tokenizing some punctuation—such as periods and commas—the resulting text seems to be more natural.)

The next problem is figuring out how much training text is required to allow the system to generate new text. With a very small amount of training text, the result would not be very interesting. That is because of the relationship between the number of unique 2-tuples and unique 3-tuples. When choosing a new word, we need to look at the final 2-tuple in the generated text. If each 2-tuple is the beginning of only a single candidate 3-tuple—if there is a 1:1 ration between 2- and 3-tuples—then after the first two words are chosen, each additional step will simply extend the generated text with an unaltered section of the source text.

In a very short sample text, the 2- to 3-tuple ratio is likely to be very close to 1:1. As the sample text gets longer, that ratio tends to get smaller.  However, it is not only the length of the text that affects the 2- to 3-tuple ratio; it is also the complexity of the text. Repetition within a text will manifest itself more greatly in smaller portions of text than in larger portions. (Even though they are very close to one another in size, 2-tuples are smaller than 3-tuples, so they will be affected more greatly by repetition.) So a large degree of repetition will result in low 2- to 3-tuple ratio relative to the total length of the text.

It is that second point that drew me to this project. The purpose of this project was to illustrate and emphasize the amount of repetition in an individual’s writing. Therefore, with a relatively small amount of sample text—I collected probably less than 2,000 words of text—the program was able to generate new text in the voice of the original author. Unfortunately, I no longer have that sample text (and the individual in question has stopped posting in that forum).

The race report that I used as an example above has, by my count, 3,687 2-tuples and 4,962 3-tuples, for a ratio of .74. It turns out that that is sufficient to start generating new text. For example, after training on that text, I can generate samples like,

The sixth aid station how far ahead he was ahead of me. Moments later, on a downhill, and I beat the aid station asked how much time I reach Bearwallow Gap. Then I’d restock again, there is a little ball on the leaders. I can do it because I had some soup, hot chocolate, scrambled eggs. I wanted to win, so there was a race. But there were a lot of strong runners on the floor and stomped on until it’s in a little.

Or,

Which brings us to 2007. I felt like there was someone behind me, that’s just post hoc rationalization. Maybe not entirely, but after that, the runner only has the former.

Now we can try to generate something new based on some other text. Hamlet, for example. I took all of the lines that Hamlet speaks in the autonymous play. By my count, in the 11,714 words, there are 10,079 2-tuples and 13,555 3-tuples, resulting in a ratio of .74. Here are some new-and-improved lines for Hamlet:

Let me not burst in ignorance; but, sir, to set on some quantity of choice, To flaming youth let virtue be as wax, And break your own inclining? Is it a free visitation? Come on–you hear this fellow no feeling of his lands will hardly lie in this box; and there is nothing either good or bad, but in the verity of extolment, I pray. The spirit that I have of late–but wherefore I know not of? Thus conscience does make cowards of us all; And now, how cheerfully my mother looks, and on this moor? Ha! have you eyes? Could you on this, The oppressor’s wrong, to hold, as pure as snow, thou shalt not escape calumny.  Get thee to a bawd than the observance. This heavy-headed revel east and west Makes us traduced and tax’d of other nations: They clepe us drunkards, and makes passionate action. The spirit that I know, to this world goes, even in that sleep of death what dreams may come When we have seen, And in this garb, lest I should have fatted all the circumstance?

To be entirely honest, I chose Hamlet because it is a well known work in the public domain. I didn’t really think it through. That certainly looks Shakespearean to me. However, this line of inquiry has made me realize that I can’t necessarily distinguish between actual Shakespearean writing and nonsense that just sounds like Shakespeare.

So I needed something else. I found the 2010 presidential State of the Union address. By my count, in 7,328 words, there are 5,894 2-tuples and 7,649 3-tuples, for a ratio of .77—slightly higher than Hamlet. What do we get from that?

The spirit that has always been our ideals, our values that allowed us to wait for decades, and jeopardize our recovery — all of their income on student loans, and are generous in spirit. These aren’t Republican values or labor values. They’re putting more emphasis on math and science; and turns around failing schools that steal the future was anything but certain. These were the times that tested the courage of our citizens, not our ambitions. So we face big and difficult challenges. And according to the world’s cheapest solar cells or treatment that kills cancer cells but leaves healthy ones untouched.  And when we are responsibly leaving Iraq to its people. Let’s leave behind the fear and division, and I know that some in my own party will argue that we can’t afford it. You hated it. You can see how their money is being spent.

There are some interesting phrases in there. However, it would be interesting to see what happens when processing text with a lower ratio.  I combined the 2001 and 2002 State of the Union addresses to gather a total of 8,221 words. But the ratio of that sample text was the same as the ratio of the 2010 address: .77. So I combined all eight of George W. Bush’s State of the Union addresses. The word count rose to 40,377, and the ratio fell to .64 (24,697:38,632).

Our shared responsibilities extend beyond matters of life and science, and that is precisely what America is the Mayor of Philadelphia. He pursued chemical, and defeatism that refuses to acknowledge that we face a tax increase they do not expect and will not be finished on our country. This year we will see freedom’s victory. In my budget provides more than a third of the people of Egypt have voted in a long way off. But we did nothing to invite. We choose freedom and security reforms. The principle here is greater security in retirement accounts. We’ll make sure that that growth is slowing. So the United States and the peace, the unfair tax on marriage will go to court because they will be effective only if it includes commitments by every major economy and add new jobs, so I ask Congress to pass these measures. I welcome the bipartisan enthusiasm for spending discipline in Washington, D.C.  Opportunity Scholarships you approved, more of America’s military families serve our country by strengthening math and science… bring 30,000 math and science.

It appears that this text bounces around more than the text with a higher ratio. Of course, now that I have a handy little PERL script (which only ended up being about 50 lines of code) to generate text, and some measure of the fitness (the 2-tuple:3-tuple ratio) of the sample text for producing interesting generated text, the next step will be to do some analysis to quantify the “interestingness” of the generated text, and to relate that to the 2-tuple:3-tuple ratio. However, that will have to wait for another day and another post.

In case anyone has interest, the script can be found here.

 

nonfiction

For Dumb Blondes?

This is Martha’s “good” conditioner. She bought it at a tony salon in town.

Apparently, the employee who was responsible for labeling the product took editorial liberties.

Back