Silician Sketchbook

Apparently, light snow soils pants. Imagine what will happen when we get our first snow storm.

I mean, come on, we haven’t even had a snow storm this fall/winter season, and people drive like 10 cm is already on the ground and quickly accumulating.

Let’s make one thing clear: the roads were not icy, they were not slushy, they were and are wet. When it rains I still see people doing 10-15 km/h above the limit, i.e. PRO DRIVER coming through. I guess they’re not so PRO if they curl up into a fetal position when they see snow that melts on contact with the pavement.

I had to take side roads, it was so bad. And the funny thing was, the side roads were very lightly traveled. I guess panicking drivers like to herd together and crawl together. When the snow really hits the fan, I’d like to see how they handle the local hill feature at their paltry 2 km/h and truly poor traction conditions.

This is the second or third time we’ve seen snow in the past few weeks.  Maybe people’s attention spans are short.

Even people in a Skype channel are treating this like it’s A Big Deal. Snow tires means driving like it’s summer? Seriously? I drove around like it’s summer in my all-seasons today. Puhleaze. I refrained from comment except to note that my winter tires continue to reside in the basement, where they have sat for the past two winters.

Maybe I ignore common sense at my own peril, but my winter tires and I don’t really get along. For one, they’re heavy, and it shows every time I have to accelerate from a stop or even take my foot off the accelerator for a moment. Mileage takes a hit as a result. Perhaps newer winter tires drive better, but the ones I have certainly do not turn a winter wonderland into a sun-kissed summer.

Most importantly, they don’t help me. They could help you, but they certainly don’t help me. I have narrowly avoided close calls in slushy to icy conditions not by coming to a stop, but by pulling quickly to the shoulder. These tires stand no chance with black ice, absolutely none. I’d rather drive my sub-optimal all-seasons and know what I’m getting into rather than think I’m king of the road with winter tires.

For those who drive around the city anyway, it’s not hard to stay out of trouble with all-seasons. Proper tire pressure, slow application of throttle, low torque/gearing for starts and stops, is that’s required from an operational standpoint. Good visibility and safe following distance are needed whether you have winter tires or not.

And if the weather is really bad, or if you know that it’s going to get really bad, then stay home. Call in sick. If you really have to be somewhere that’s not the airport, take the bus, a vehicle that’s guaranteed to have more traction than you’ll ever get out of your winter tires.

Piled Higher and Deeper is a very QFT worthy comic strip, but this one in particular reminds me of what my macroeconomics professor said during one lecture.

Graduating? Economy going to tank? Stay in school, anyway you can.

The rationale was that job prospects, while not non-existent, were going to be harder to come by, and when the they pick up you will be in competition with younger graduates with current experience versus whatever you received 2+ years prior.

Ninja Edit: Nice follow.

When one of the channels on a headset drops out, and you have to fiddle with the wires to find the sweet spot such that it’ll come back, you know that the headset is on its last legs. But I’m not about to go and buy a new headset just yet. The one I was using was not really mine at first, and I wound up claiming it because no one else was using it.

I’ve since gone back to my original headset, but there was a reason why I stopped using it in the first place, which I’ve just rediscovered.

For one, the foam piece that covered the microphone spontaneously came off, but that’s not a big deal. More troublesome was that one day, the headset decided to switch to mono, and at the time I couldn’t figure out why.

A cool side-effect of having the headphones magically go mono is that the vocal channel of a song is either heavily attenuated or outright eliminated. And it turns out that if I twist the headphone connector around enough, I can likewise ditch the karaoke channel and get the vocals.

The novelty doesn’t last very long, but fortunately I can get stereo back by pulling the connector out slightly. One side is slightly attenuated as a result, but it’s better than being completely blindsided in a game.

I was late in getting home last night, like pretty much everyone else. I was one of those thousands of people that crowded Yonge and Bloor, although I wasn’t so silly as to step into the middle of the street and block traffic and shuttle buses.

Times like these, it’s good to not have tunnel vision (the irony). I was entertaining the idea of slipping back into the station, doubling back to the Spadina line via the Bloor-Danforth trains, and going north to Lawrence West station. I could then head east to Lawrence station where the subway was still operating northbound. A lot of people did wind up doing something similar, and it’s a shame that more didn’t.

I, on the other hand, took the Bloor line east instead, because east was where I needed to be anyway. I haven’t been on the Bloor line in a while, and the sad truth is I’ve been spoiled by the Yonge-University line’s speed and longer inter-station distances. I’m not sure how anyone could be spoiled by Yonge-University, but there you go.

Opinions vary, but a TTC subway extension northbound was needed yesterday, or 5+ years ago (and don’t get people started about the extension to York University).  If we’re lucky, we’ll see an extension to the Yonge line north to Highway 7 in eight years (see second last page), of which only five are actual construction.

And if you look at page 7, you’ll notice that this extension is only 6.5 kilometers. So 6.5 km in 8 years. If you believe in Wikipedia, this is something amounting to an improvement over the last extension — the Sheppard line — which was 5.5 km in 8 years.

I don’t know about you, but this seems downright atrocious. To put things in a bit of perspective, construction on the Chunnel was completed in 6 years. So in the time it takes to extend the Yonge line 6.5 km, plus one more year, others have built roughly 50 km of track underwater. Should we, perhaps, be embarrassed?

Okay, so maybe that was a bit of an apple to oranges comparison (and only because the Chunnel is way cooler than a middling subway extension). Comparing subways to subways, in the time it would take to construct our subway extension, the Chinese built an entire line. Line 5 of the Beijing Subway, to be exact, spanning 23 stations and 27.6 km.

In 5 years, the Chinese have also built the first phase of Line 10, and this phase alone is 22 stations and 24.68 km in length.

I mean, come on.

But if there’s a silver lining to all this, is that in their latest expansion, the STM took 5 years to extend their Orange Line 4 km and build 3 new stations. So I guess we can laugh at them for being slow?

So last night, my ISP kicked me off the internet for the rest of the day. Of this, there is little doubt, because some time after midnight, service resumed.

I felt compelled to consume as much of my remaining monthly quota as possible, and I think they caught on. It wouldn’t surprise me if others were attempting the very same thing. I was approaching 20 GB of total traffic when things just ground to a halt. I rebooted my router, I power cycled my modem. No dice.

Occasionally, there would be a burst of speed so I could load the rare web page, but for the most part I was pretty much being denied service. If you squint really hard, you could call it throttling.

Lesson learned: start the spree in advance, and limit to 15 GB days to be on the safe side.

Yeah, I’m one of them.

Usually if you’re coming into error correcting codes with no prior background (and parity bits doesn’t cut it), it’s best to just get a book on the topic. There aren’t that many references on the web that just tell you how to do it. A lot say what to do, but fail to provide details. To some extent, most times the details are boxed up in intellectual property cores or libraries so you don’t have to worry about them, but I was looking for at once a somewhat leaner and more specific implementation, plus a bit of education.

I found one that does a good job at both. Searching around eventually led me to a BBC white paper on Reed-Solomon coding that amounted to a gentle overview of the theory plus implementation.

Along with a couple references for multiplication over GF(256), I first wrote a multiplier function, and then used that to construct a 64 KB multiplication table. To all you veterans who frown at this seemingly inefficient approach, 64KB in memory is nothing when you have 2 GB it.

Now I’ve just got to write the rest of the forward error correction stack, and the subsequent decoder for it all.

So what would compel me to fish out my clarinet from underneath a desk, 7 years since having last closed it? Nothing short of a surprise performance to pay tribute to a retiring high school music teacher. That is to say, retiring from teaching high school music, but “graduating” to a university music program. I’m not too inclined to name names — got to keep this place marginally anonymous — but Google is really a click away.

I think that, if you were going to show up to play, then you likely spent at least a good 4 years playing one instrument, and from what I saw from the only rehearsal the day before the event, that kind of experience is hard to lose. The moment the band attempted its first run at Lyric Essay (Donald Coakley) after tuning, there was a very real sense that we could pull this stunt off.

Us alumni had help. The thing that goes first with lack of practice is technique, and nothing that we played was too demanding overall, and what tricky spots there were could be faked since it wasn’t a prominent section. There were quite a few people who continued music through university, several of them becoming music teachers in their own right, and we were also supplemented by current high school music students in parts where we were understaffed.

Most had the benefit of seeing these pieces before, although it was mostly sight-reading for me. We performed Lyric Essay, an abridged version Children’s March (Percy Grainger), and Ross Roy (Jacob de Haan). I had never seen the first, I had played a different (and easier) arrangement of the second in the past, and my memory is fuzzy on the third. It’s a grade 10 piece, or advanced grade 9, and I may have played it in class, but I don’t think it was something I played in serious rehearsal or concert.

I took a demotion to second clarinet, but the firsts continued to play after they graduated, whereas I clearly had not. Took for one for the team, and got to not embarrass myself to boot.

Saturday night was largely successful. Completely surprise was achieved, no mean feat considering that the entire music department and the wife was in on it. There was an elaborate story to drag him into the school involving a friend with a broken down car, and a non-existent volleyball game as an excuse for all of the cars in the school parking lot.

But our first piece of the night, Children’s March was shaky In the run-up to the evening, we held a quick half-hour refresher and played through it, and I was stunned to hear the band fall apart in the middle. At least a third of us got lost in one of those cascading effects where people came in when they weren’t supposed to.

There was a disturbing drop in volume as people took the most appropriate action and waited for a landmark section to come back in, and we recovered. It didn’t happen the night before, but it happened again during the real thing. It might have been a combination of nerves and the addition of new students, as there were a bunch of faces that I hadn’t seen during rehearsal. Anyway, two scary moments.

We managed to get our act together for a “guest of honour” conduction of Lyric Essay, and it went off without incident, being not technically demanding and all. In between pieces, staff, former staff (plus former head of the music department), former students, and friends all went to the microphone to speak. At turns touching and hilarious, the night was a class act, the same as was demanded in the past.

Ross Roy was freaking amazing, or “not bad for 80-something people playing together for the second time in two days.” Afterwards, he said that it was expected. After all, we were students of great bands. I think he might have a point.

But if nothing else, winging it is a lot of fun.

Except I can’t handle hills.

Prior to Thursday, I hadn’t opened my clarinet case in 7 years. I know that the past couple days I kept saying 6, but neglected to remember that I fast tracked. Despite the age, it was in the condition as I remembered it, with the exception of really stiff cork tendons on all of the parts. Even the reeds in the case hadn’t aged. At all. I hadn’t exactly kept my clarinet case in a cool and dry place, but I suppose that it was both cool and dry enough, and the case afforded some additional margin, and my clarinet itself is plastic.

Don’t boo me or anything. I was a competent player, but a career in clarinet never really crossed my mind. Now that I cared to look at my clarinet’s make and model, turns out it’s a Vito Reso-Tone 3, one of the better known plastic clarinets. It’s second-hand, purchased approaching 11 years ago for $250 at the time, about what it would have cost to rent the same clarinet for 2 years. It could use some reconditioning now, or even 7 years ago, but I’m not sure when I’m going to play it again, if ever.

This is a follow-on from my previous post. I mentioned how I wrote my own app after giving up on Yahoo Pipes’ Unicode quirks. Well, I think I found the solution. It’s partly Yahoo’s fault, and partly me not reading some documentation.

I consider myself a programming novice in general, not just in Python, so bear with me.

On Yahoo’s side, it has inconsistent support for UTF-8. On the pipe output there is a mix of proper UTF-8 rendered characters, as well as wrong characters stemming from each byte of their multi-byte sequences being placed as-is.

Here’s a common example that I came across. Consider the byte sequence 0xE28099, typically represented in Python as “\xe2\x80\x99″. When interpreted as a UTF-8 character, it winds up as ‘U+2019′ or in Python as u\’u2019′. The u prefix indicates a unicode string, and the escape sequence basically says Unicode character 2019. If you go to print this out, it is the right single quote, i.e. an apostrophe.

But rather than rendering an apostrophe in the pipe output, you get “\xe2\x80\x99″ but in a Unicode string. So all over the place I see things like, “I\xe2\x80\x99m” and “haven\xe2\x80\x99t”. Raw UTF-8 embedded in UTF-8. Fun stuff. To my knowledge, there’s no operator within Pipes that could fix something like this, although there might be a regex that could do the trick. I don’t know the first thing about regular expressions though.

Not too long before, while looking for articles on Google App Engine, I came across a post on using GAE for the Web Service operator. Free hosting for Pipes filters? Sign me up!

Yahoo Pipes posts its data to the address in the Web Service operator, in the form of a UTF-8 encoded JSON string. It expects either a JSON string or RSS/XML document in return, but it’s probably more convenient to return JSON since you’re working with it. Plus, I did try the RSS/XML option once but Pipes output just showed a zero. I didn’t feel like pursuing the problem.

So here is the handler that services the Yahoo Pipes request:

from django.utils import simplejson

class PipesScrubber(webapp.RequestHandler):
  def post(self):
    data = self.request.get('data')
    cleaned = tidy_unicode(data)
    posts = simplejson.loads(cleaned)['items']
    self.response.headers['Content-Type'] = 'application/json'
    simplejson.dump(posts, self.response.out, ensure_ascii=False)

The simplejson library is bundled with django which is bundled with GAE so you don’t have to upload your own. What I neglected to do the first time I tackled this problem was to set that ensure_ascii argument to False. Looking back, it would be the obvious thing to do because if I got UTF-8 JSON, I should return UTF-8 JSON.

But by default, simplejson generates ASCII-encoded strings, while escaping all HTML and Unicode characters. I was baffled by this for the longest time, because the solution seemed sound but the Pipes output had just as strange looking stuff because of all the escaping. Turns out I should have read the simplejson documentation a bit more thoroughly.

Here is the tidy_unicode function that does all of the heavy lifting:

import StringIO
import array

def tidy_unicode(raw):
  ret = StringIO.StringIO()
  lim = len(raw)
  i = 0
  while i < lim:
    try:
      val = ord(raw[i])
      if val < 0x80 or val > 0xF4:
        ret.write(raw[i])
        i += 1
      elif 0xC0 <= val <= 0xDF:
        s = array.array('B', [ord(c) for c in raw[i:i+2]]).tostring()
        ret.write(s.decode('utf_8'))
        i += 2
      elif 0xE0 <= val <= 0xEF:
        s = array.array('B', [ord(c) for c in raw[i:i+3]]).tostring()
        ret.write(s.decode('utf_8'))
        i += 3
      else:
        s = array.array('B', [ord(c) for c in raw[i:i+4]]).tostring()
        ret.write(s.decode('utf_8'))
        i += 4
    except UnicodeDecodeError:
      raise NameError(array.array('B', [ord(c) for c in raw[i:i+4]]))

  return ret.getvalue()

The try/except block is left-over debug code when I was hitting my head against my desk. Doesn’t hurt to leave it in, as the JSON strings aren’t super long so performance isn’t that much of a concern anyway. Which is good, because this code is quite horrible, like trying to write C in Python and expecting the same performance. But I couldn’t think of any other way (always open to suggestions though). After all, the problem is that raw UTF-8 strings are being embedded in an otherwise fine UTF-8 string.

As such, you can’t just do something like utf_8_str.decode(’utf_8′) because it would treat each “raw” byte as ASCII, find that the leading byte is greater than 127, and then blow up. Instead, the byte sequences have to be isolated and converted to a regular string which can be interpreted as UTF-8. Complicating matters a tad is that byte strings are variable length, so range checks have to be performed to calculate the length.

One thing that could be changed right off the bat, besides removing the try/except block, is the use of generator expressions instead of list comprehensions when doing the byte sequence conversion. Might be a bit more memory efficient, but on GAE memory is dirt cheap, and this app isn’t using gobs of it anyway.

There you have it. Hope it was useful, or educational at least.