Spacefem (spacefem) wrote,

beginner's guide to information theory

I'm feeling better tonight. last night I did very poorly on an exam, hence the bitchiness. Today I sat down and looked at some numbers, and realized that since I did pretty well on the first two tests, last night's score (plus the final) don't really mean all that much. I have to average a 55% to keep my B. I could all but not show up and still get a C... which is really what matters, since my company requires a C or better to repay my tuition (the going phrase around the office is "C is for Cash").

With that out of the way, I would like to write a little about what I've been studying this semester: Information Theory. Because even though I hate school right now, a little part of me deep down inside is appreciating the material and even sometimes thinks it might be cool. And who knows, someone else here might think the same thing... I keep trying to explain it all to people, and some of them just immediately smile that "please stop" smile, tell me they can't even begin to understand it, and compliment me for being smart enough to even get past the first chapter. This isn't really a compliment... being smart enough to understand brilliant ideas is pointless if you're not capable of explaining it in layman's terms to anyone new off the street. But I figure maybe it's not me, they just don't want to understand it... math phobia? Other people, mostly ones who are uncomfortable admitting they don't get something, hear me out on it and eventually agree that it is kind of cool. So here's hoping my lj readers are those kinds.

Information theory starts with the idea that we transmit symbols using bits (example, A = 01000001 B = 01000010) because computers talk using 0s and 1s. On a transmission line, we can represent 0s and 1s with things like frequencies or voltage levels (a certain level means 0, another one means 1) or whatever, but either way each bit takes a little bit of time to transmit. You have to send the "1" signal for a while, and the "0" signal for a while, and give the guy on the other end time to process it. Or you're storing these 0s and 1s in a file, that takes up space on your hard drive, and that's annoying too.

So if you start looking at your signal, and notice that certain parts of it repeat a lot, you assign shorter bit sequences to those parts (example, A = 1, B = 0001). Rather than grouping eight bits in a row together every time to figure out what letter you just sent, the receiver at the other end looks for those unique patterns. Huffman Coding is a cool example of this... the wiki explination looks sort of crazy so maybe I'll take that one on in another entry.

Anyway there's an expression we use for the uncertainly of a signal called "entropy" (not like the thermodynamics entropy, in case someone here is into that). If I'm sending you a message, and every single letter happens the same number of times, then you have no way to predict what's going to come next. But if I just send you mostly As and the occasional B, then you can be pretty certain that the next letter you'll get is A. More certainty = less entropy, and less entropy means you can get more out of all this information theory stuff. So we look at ways to reduce all that by using combinations of symbols together (ie, U generally follows Q, we can use that to help us out).

So that was the first day of class, and the rest of the semester was spent looking at probability distribution functions (which are pretty terrible and should be avoided) and how we can use what we know from them to use less bits to transmit the same set of data, all the time, with no distortion or loss.

And in two weeks the class will be over and I will forget all this, and part of me can't wait. you know?
  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded