A Day In the Office

So I'm talking to one of my co-workers when he mentions his cousin has roid rage and another family member has Aspergers syndrome. I offer that I have hemorrhoids too. He looks at me funny. Then he says he has AADD. I ask what that is and he says its adult attention deficit disorder. I tell him I had that too but now its cured. He asks what cured me and I tell him my dad smacked me in the head and told me to 'pay attention', so he gets up and leaves.

A few of the office secretaries come to my table and sit down. They give me the quick look over and decide, I guess, that I am safe enough, being almost 30 years older than them so they start talking about sex to each other. I interrupt and say, "don't talk about sex in front of me, I'm a married man - I haven't had sex in like 5 years". They all laugh, so I get up and see if I can find a less rude, more understanding table.

I sit down with some younger guys and we start talking about real estate. One of the guys asks "hey Ken, how many properties do you own". Not wanting to sound like a big shot I say, well, I just have an itty bitty house up in Central Florida, a small little condo down in South Florida with two little tiny dogs. One of the kids asks "gee Ken don't you have anything big" and I answer "yea my wife" at which point they all stare at me so I decide to move to yet another table.

I'm sitting alone when my boss comes up and sits next to me. "How many hours do you have into this project so far" he asks. I reply "I have goals on my mind, not hours" to which he responds, "I have goals too and one of mine is to see your hours". I try the water cooler.

Just as I'm getting ready to take a drink of water a guy comes up to me and stops me. "Hey don't drink that" he warns, "do you know what fish do in that?". He hands me a beer. I look at him and say "it's like 8:30 in the morning, I don't drink before 9am". He laughs and takes a swig from his. Finally, a kindred spirit.

Favorite Cuss Word As Of Today

I believe my favorite cuss word lately, that is, a word I can't use in mixed company, is fucktard. Its a concatenation of the words 'fucking' and  'retard', and it is one of my favorite 'dirty words' lately because it is so applicable to what I have had to deal with over the last year; fucktards.

Fantasy Football 2012 Team One

THE DRAFT

So I drafted my first team July 12th at 9pm. Its a standard $350 NFFC On-Line with (12 team snake draft with third round reversal) a top prize of $50k. I felt that rather than just discuss a bunch of different players I would simply comment on why I drafted who I did where.

I drafted from the ninth position which is exactly what I had asked for in my pre-draft preferences. My good friend Jimmy, who is also in this league, had asked for the sixth spot and he got that too. Just an FYI - in this league before the draft you set your draft position preferences from spot one to twelve.

 My rationale for drafting ninth is that there are seven really good running backs (RBs) this year (Arian Foster, Ray Rice, Lesean McCoy, Maurice Jones-Drew, Ryan Matthews, Chris Johnson and Darren McFadden) who to me are pretty much the same, or at least I wouldn't mind drafting any one of them first, so drafting seventh would be better than picking first since I would then get a better second round pick. Now I also figure both Calvin Johnson and Aaron Rodgers will be gone by the ninth pick so that means I will get one of the previously mentioned nine players at the ninth spot, who to me, are all worthy of a first round pick and just about equal in value. Of course many will nit pick and say "McFadden, really?" or "Aaron or Calvin are a hell of a lot more valuable then Chris Johnson", but I feel getting any one of the aforementioned at the ninth spot is good value.

Given a third round reversal the ninth spot gets three really good picks (ninth, fourth and fourth) so I was hoping to get a RB in the first round and then another RB in the second. If a really good one fell to me, I would also take one in the third round, otherwise I take a WR. I think there are about 20-24 decent RBs this year then the quality falls off drastically. Not so with wide receivers. There are tons of them this year. So my general draft strategy for this year is to pick from the number nine spot and pick RB-RB-RB-WR-WR=WR=QB-TE-QB.

All of this strategy is subject to my one basic fantasy draft rule which is that you take the best available player regardless of position with the first three picks, with a minor addendum being don't take two QBs or TEs with the first three picks. So, I am always going to adapt my draft strategy if I see a player I feel is a number one pick, for example, available in the third round. So, going into my first draft, this is kind of how I was thinking.

Here is how the draft actually went with my picks bolded ...

1 1 Arian Foster RB 1 2 Ray Rice RB 1 3 LeSean McCoy RB 1 4 Calvin Johnson WR 1 5 Aaron Rodgers QB 1 6 Ryan Mathews RB 1 7 Tom Brady QB 1 8 Chris Johnson RB 1 9 Maurice Jones-Drew RB 1 10 Trent Richardson RB 1 11 Drew Brees QB 1 12 Matthew Stafford QB

2 1 Wes Welker WR 2 2 Darren McFadden RB 2 3 Jimmy Graham TE 2 4 Marshawn Lynch RB 2 5 Andre Johnson WR 2 6 Rob Gronkowski TE 2 7 Larry Fitzgerald WR 2 8 DeMarco Murray RB 2 9 Roddy White WR 2 10 Matt Forte RB 2 11 Cam Newton QB 2 12 Jamaal Charles RB

3 1 Brandon Marshall WR 3 2 Dez Bryant WR 3 3 Darren Sproles RB 3 4 A.J. Green WR 3 5 Julio Jones WR 3 6 Greg Jennings WR 3 7 Hakeem Nicks WR 3 8 Adrian Peterson RB 3 9 Mike Wallace WR 3 10 Steve Smith WR 3 11 Victor Cruz WR 3 12 Steven Jackson RB

 4 1 Frank Gore RB 4 2 Percy Harvin WR 4 3 Brandon Lloyd WR 4 4 Miles Austin WR 4 5 Marques Colston WR 4 6 Jeremy Maclin WR 4 7 Jordy Nelson WR 4 8 Demaryius Thomas WR 4 9 DeSean Jackson WR 4 10 Dwayne Bowe WR 4 11 Kenny Britt WR 4 12 Fred Jackson RB

5 1 Doug Martin RB 5 2 Ahmad Bradshaw RB 5 3 Michael Vick QB 5 4 Peyton Manning QB 5 5 Jahvid Best RB 5 6 Michael Turner RB 5 7 Vernon Davis TE 5 8 Antonio Brown WR 5 9 Tony Romo QB 5 10 Eric Decker WR 5 11 Aaron Hernandez TE 5 12 Vincent Jackson WR

 6 1 Stevie Johnson WR 6 2 C.J. Spiller RB 6 3 Antonio Gates TE 6 4 Jason Witten TE 6 5 Jermichael Finley TE 6 6 Roy Helu RB 6 7 Reggie Bush RB 6 8 James Starks RB 6 9 Shonn Greene RB 6 10 Torrey Smith WR 6 11 Jonathan Stewart RB 6 12 Willis McGahee RB

7 1 Reggie Wayne WR 7 2 Fred Davis TE 7 3 BenJarvus Green-Ellis RB 7 4 Robert Meachem WR 7 5 Philip Rivers QB 7 6 Isaac Redman RB 7 7 Eli Manning QB 7 8 DeAngelo Williams RB 7 9 Beanie Wells RB 7 10 Matt Ryan QB 7 11 Stevan Ridley RB 7 12 Justin Blackmon WR

8 1 Michael Crabtree WR 8 2 Mark Ingram RB 8 3 Donald Brown RB 8 4 Jacob Tamme TE 8 5 Greg Little WR 8 6 Denarius Moore WR 8 7 Pierre Garcon WR 8 8 Darrius Heyward-Bey WR 8 9 Tony Gonzalez TE 8 10 Peyton Hillis RB 8 11 Santonio Holmes WR 8 12 Brandon Pettigrew TE

9 1 Lance Moore WR 9 2 Ben Roethlisberger QB 9 3 Jermaine Gresham TE 9 4 Danny Amendola WR 9 5 Sidney Rice WR 9 6 Malcom Floyd WR 9 7 Ben Tate RB 9 8 Toby Gerhart RB 9 9 Felix Jones RB 9 10 Ronnie Hillman RB 9 11 Titus Young WR 9 12 Jay Cutler QB

 10 1 Dustin Keller TE 10 2 Brent Celek TE 10 3 Michael Bush RB 10 4 Pierre Thomas RB 10 5 Mike Williams WR 10 6 Ryan Williams RB 10 7 San Francisco DST 10 8 Jared Cook TE 10 9 Robert Griffin III QB 10 10 David Wilson RB 10 11 Doug Baldwin WR 10 12 Shane Vereen RB

 11 1 Austin Collie WR 11 2 Jacquizz Rodgers RB 11 3 Isaiah Pead RB 11 4 Daniel Thomas RB 11 5 LeGarrette Blount RB 11 6 Mikel LeShoure RB 11 7 Greg Olsen TE 11 8 Nate Washington WR 11 9 Kevin Smith RB 11 10 Randy Moss WR 11 11 Anquan Boldin WR 11 12 Michael Floyd WR

12 1 Matt Schaub QB 12 2 Alex Green RB 12 3 Jake Locker QB 12 4 Carson Palmer QB 12 5 Owen Daniels TE 12 6 Joe Flacco QB 12 7 Ryan Fitzpatrick QB 12 8 Santana Moss WR 12 9 Laurent Robinson WR 12 10 James Jones WR 12 11 Mike Goodson RB 12 12 Delone Carter RB

13 1 Andrew Luck QB 13 2 Eddie Royal WR 13 3 Brandon LaFell WR 13 4 Chad Johnson WR 13 5 Josh Freeman QB 13 6 Vincent Brown WR 13 7 Jon Baldwin WR 13 8 Coby Fleener TE 13 9 Tim Hightower RB 13 10 Nate Burleson WR 13 11 Andy Dalton QB 13 12 Alshon Jeffery WR

 14 1 Kyle Rudolph TE 14 2 Jerome Simpson WR 14 3 Randall Cobb WR 14 4 Bernard Scott RB 14 5 Mike Tolbert RB 14 6 Jacoby Ford WR 14 7 Earl Bennett WR 14 8 Chicago DST DST 14 9 David Nelson WR 14 10 Kendall Wright WR 14 11 Davone Bess WR 14 12 Evan Royster RB

15 1 Heath Miller TE 15 2 Joe McKnight RB 15 3 Taiwan Jones RB 15 4 Sebastian Janikowski K 15 5 Rashard Mendenhall RB 15 6 David Akers K 15 7 Rashad Jennings RB 15 8 Philadelphia DST 15 9 Brian Quick WR 15 10 Ed Dickson TE 15 11 Rueben Randle WR 15 12 Green Bay DST

16 1 Detroit DST 16 2 Houston DST 16 3 Buffalo DST 16 4 Baltimore DST 16 5 Dexter McCluster RB 16 6 New York (N) DST 16 7 Jason Snelling RB 16 8 New York (A) DST 16 9 Pittsburgh DST 16 10 Seattle DST 16 11 New England DST 16 12 Atlanta DST

 17 1 Dallas DST 17 2 Stephen Gostkowski K 17 3 Devery Henderson WR 17 4 Sam Bradford QB 17 5 Lance Kendricks TE 17 6 Brian Hartline WR 17 7 Dan Bailey K 17 8 Arizona DST 17 9 Cedric Benson RB 17 10 Andre Caldwell WR 17 11 Emmanuel Sanders WR 17 12 Mohamed Sanu WR

18 1 Knowshon Moreno RB 18 2 Mason Crosby K 18 3 Kendall Hunter RB 18 4 Javon Ringer RB 18 5 Golden Tate WR 18 6 Lamar Miller RB 18 7 Kellen Winslow TE 18 8 Chris Rainey RB 18 9 Dallas Clark TE 18 10 Montario Hardesty RB 18 11 Joseph Addai RB 18 12 Jordan Shipley WR

 19 1 Matt Prater K 19 2 Martellus Bennett TE 19 3 Rob Bironas K 19 4 Ronnie Brown RB 19 5 Robbie Gould K 19 6 Kevin Walter WR 19 7 Tim Tebow QB 19 8 Matt Bryant K 19 9 Bernard Pierce RB 19 10 Alex Henery K 19 11 Steve Breaston WR 19 12 Chris Givens WR

 20 1 Garrett Hartley K 20 2 Nick Toon WR 20 3 Leonard Hankerson WR 20 4 Jason Hanson K 20 5 Alex Smith QB 20 6 Tony Moeaki TE 20 7 Lance Ball RB 20 8 Robert Turbin RB 20 9 Dion Lewis RB 20 10 Matt Flynn QB 20 11 Kellen Davis TE 20 12 Marcus Easley WR

MY ANALYSIS

I was very happy MJD dropped to me at the ninth pick. I consider him the 4th or 5th best back in the draft. I assume his current hold-out helped. I was hoping McFadden would fall to me in the second but when he went it was a no-brainer taking Lynch, and keep in mind I took him before he got his DUI. I would probably have taken him even if I knew about the DUI. I suspect he will have a big year in Seattle.

 I was very conflicted in the third round but ended up taking AJ Green over a third running back. I absolutely love Green and consider him the second best receiver in the draft just ahead of Larry Fitzgerald. He had more down field targets last year (attempts over 20 yards or maybe its 30) than anybody, if I'm not mistaken, and he should get better in his second year. He is one of the few rookie WRs or RBs I can think of who have made an impact in their first year.

My fourth round pick was a bit long in coming due to the nature of the third round reversal. I watched a lot of quality go before it was my turn. I ended up choosing Deean Jackson over Michael Vick and Ahmad Bradshaw mainly because he is fresh off a new contract and should be motivated this year, which I feel he was not last year. Also, he is one of the fastest WRs in the league even though he's a pretty big guy. I like him maybe even more than Percy Harvin who I was thinking I would end up with here.

My fifth pick is where I decided to take my first major risk. Peyton Manning may have a great year in Denver or he way have a terrible year and even get hurt, but I still decided at this point that I would go with Manning and RGIII; two big risk/reward QBs. If I was going to not play it safe at any position this year I had already decided it would be QB. Of course I now had to watch how the QBs went off the board and decide how long I could risk not taking Robert Grffin but it was clear I was now going to have to draft him at some point in the next 3-4 rounds.

I took Shonn Green in the 6th round mainly because other than Green-Ellis I didn't think there were any decent non-platoon starters left. I think this is the last year for Green; either he kicks ass or he's out in NY. I figure they will run the wildcat up there a lot with Tebow and Sporano so I see Green getting a ton of carries (around 275-325) in a more open offense than he has run in the past. He is a risk, but as my 3rd RB and potential flex I felt he had the upside to justify that risk.

I may have made a mistake and reached for Meechem in the 7th but WRs were getting scarce and all the news coming out of early Chargers training camp is that Norv Turner loves him and he will probably be their number one receiver this year. It is interesting that Peter King just posted an article today that said "Royal over Meechem" but I don't agree. Anyway, by this point in the draft it was pretty clear I was gonna have to take some chances to find a decent number three receiver as most of the good receivers were already gone by the 7th round.

Taking Tony Gonzalez over Brandon Pettigrew in the 8th may prove to be the biggest mistake I made in this draft. Pettigrew is Stafford's second favorite target after Calvin and I think he will only get better this year. I like Gonzalez though as I think the Falcons will be a high scoring offense this year and teams will be covering White and Jones at the expense of leaving a line backer on Gonzalez who should get his usuall 8-12 TDs this year.

The ninth round is where I went against my better judgement and drafted Danny Amendola over Sidney Rice because my son Edward kept screaming in my ear. You would think by now I was immune to such behavior but let's just say this was not really my choice though I do agree Amendola has an opportunity to catch over 60-70 passes this year. He could be a poor man's Welker so I am not terribly upset over this pick, unless of course Rice has a big year in Seattle which is what I am expecting (note - Ed says he is coming off shoulder surgeries and may not be the same player he was before the operations).

When I saw Jay Cutler go off the board at the end of the ninth I knew it was time to pull the trigger on Griffin. I actually was happy to get him in the tenth as I thought I might have to draft him earlier.

I took Daniel Thomas in the 11th round and was very happy with him, though my friend Jimmy who drafted sixth thinks he will be the 3rd back after Bush and Miller. He is high on Lamar Miller but I still think Thomas will be the starter before the season ends (unless Reggie Bush does not get hurt which I think is highly unlikely). Miller does have some wheels though. I'm pretty sure he wiull be returning kickoffs for the fish this year.

I took Laurent Robinson in the 12th round because he was a big free agent acquisition for Jacksonville and I know they will target him a lot. In fact I believe he will get about twice as many targets as Blackmon and thats if Blackmon can keep out of trouble. I also am banking on Chad Henne taking over the starting QB spot, hopefully, sooner rather than later. He is a NFL caliber QB even if his time in Miami did not show it. In time I expect either Robinson or my next pick to become my third WR.

Chad Johnson (once known as Ochocinco) was available in the 13th round so I took him. Even though his coach said "he will be our number one wide receiver if he makes the team" I just don't think the Dolphins have any choice at this point. Devon Bess is a decent slot receiver but he is not a number one WR nor is he much of a down field threat. Ochocinco said the Patroits kept him down last year and I believe it. I think he will have a bounce back year this year, my only concern being who will throw him the ball? I'm thinking Garrard for half the season, Tennehill for a few games then Matt Morre. Its a mess which is why Ocho is a 13th round pick. I'm keeping my fingers crossed.

I have absolutely nothing to say about my 14th round pick, David Nelson, except that my son Edward basically made it. He loves him, and I agreed to take him so we'll see. Had I not taken Nelson here, I would have taken Evan Royster or Rashard Jennings (MJD's handcuff).

Janikowski and Baltimore are two sentimental picks for me as I have had both on just about every money team I've had over the years and they always do well.

Ronnie Brown and Dallas Clark were my picks but Deion Lewis (my last pick) was made by Ed. I suspect Ronnie Brown and Deion Lewis will be on the waiver wire by week five but you never know. Dallas Clark might have been the biggest sleeper of the draft now that Winslow is in Seattle. I felt he was a great risk at this point in the draft.

There Was A Serpent

There was a serpent and man did smite him by cutting him in half, however, the serpent refused to die and instead man now had two serpents to deal with, so man did devise a method to smash the two serpents into a million tiny little pieces which he did and then man had a million tiny little serpents to deal with until finally man decided to give up at which point the serpents died due to lack of attention.

So I think I am going to blog my 2012 fantasy football season. I will post things which were important to my decision making process and I will discuss my rationale for various draft picks I make. I am typically an early starter and this year is no different. I have started with the Vegas odss for the teams for the upcoming season. The Vegas odds I found on-line as of June 1st 2012 are as follows ...

Green Bay Packers +$500 (5 to 1)
New England Patriots +$500 (5 to 1)
Philadelphia Eagles +$800 (8 to 1)
San Francisco 49ers +$800 (8 to 1)

Houston Texans +$1,100 (11 to 1)
Baltimore Ravens +$1,200 (12 to 1)
Denver Broncos +$1,200 (12 to 1)
New Orleans Saints +$1,200 (12 to 1)
New York Giants +$1,300 (13 to 1)
Pittsburgh Steelers +$1,600 (16 to 1)
Chicago Bears +$2,400 (24 to 1)
Atlanta Falcons +$2,500 (25 to 1)
Dallas Cowboys +$2,500 (25 to 1)
Detroit Lions +$2,500 (25 to 1)
New York Jets +$2,500 (25 to 1)
San Diego Chargers +$2,500 (25 to 1)
Cincinnati Bengals +$3,000 (30 to 1)

Kansas City Chiefs +$4,000 (40 to 1)
Buffalo Bills +$4,000 (40 to 1)
Carolina Panthers +$4,000 (40 to 1)
Arizona Cardinals +$5,000 (50 to 1)
Tennessee Titans +$5,000 (50 to 1)
Oakland Raiders +$5,000 (50 to 1)
Miami Dolphins +$5,000 (50 to 1)
Seattle Seahawks +$6,000 (60 to 1)
St. Louis Rams +$8,000 (80 to 1)
Washington Redskins +$8,000 (80 to 1)
Cleveland Browns +$10,000 (100 to 1)
Indianapolis Colts +$10,000 (100 to 1)
Jacksonville Jaguars +$10,000 (100 to 1)
Minnesota Vikings +$10,000 (100 to 1)
Tampa Bay Buccaneers +$10,000 (100 to 1)


This is where I will begin my analysis. Well, with one additional bit of information. Here is how the teams finished last year

NFC
NFC EAST W L T PCT
NY Giants 9 7 0 .563
Philadelphia 8 8 0 .500
Dallas 8 8 0 .500
Washington 5 11 0 .313


NFC NORTH W L T PCT
Green Bay 15 1 0 .938
Detroit 10 6 0 .625
Chicago 8 8 0 .500
Minnesota 3 13 0 .188


NFC SOUTH W L T PCT
New Orleans 13 3 0 .813
Atlanta 10 6 0 .625
Carolina 6 10 0 .375
Tampa Bay 4 12 0 .250


NFC WEST W L T PCT
San Francisco 13 3 0 .813
Arizona 8 8 0 .500
Seattle 7 9 0 .438
St. Louis 2 14 0 .125


AFC
AFC EAST W L T PCT
New England 13 3 0 .813
NY Jets 8 8 0 .500
Miami 6 10 0 .375
Buffalo 6 10 0 .375


AFC NORTH W L T PCT
Baltimore 12 4 0 .750
Pittsburgh 12 4 0 .750
Cincinnati 9 7 0 .563
Cleveland 4 12 0 .250


AFC SOUTH W L T PCT
Houston 10 6 0 .625
Tennessee 9 7 0 .563
Jacksonville 5 11 0 .313
Indianapolis 2 14 0 .125


AFC WEST W L T PCT
Denver 8 8 0 .500
San Diego 8 8 0 .500
Oakland 8 8 0 .500
Kansas City 7 9 0 .438


Now, armed with that information, we can begin by looking to see who the Vegas odds makers think is going to do better (or worse) this year. This is a good place to start and then we can look at each case and see if we agree or disagree with them and why.

Fantasy football is an interesting thing because it is often the case that changes in non-position players has an affect on position players. As an extreme example, take Peyton Manning. If you put him on a team with 5 bad rookies on the offensive line then he will get killed, if you insteaed put him on a team with 5 all pro offensive lineman who will give him unlimited time then he could be throwing to high school players and still do well, so it is often the case that non-position players will be one of the reasons the Vegas odds makers move a team up or down from where they finished the previous year. When we later analyze the deltas the Vegas guys have predicted we will make sure we understand what is going on with the non position players before we jump to any conclusions about the position players.

Now it is also the case that rookies in the NFL, while very flashy, rarely make an impact their first year. Yes, some QBs have come through recently that blow that theory out of the water (although it goes the other way more often than not), but think about it. When was the last time a rookie WR or RB came into the league and did anything of sgnificance? Last year, A.J. Green is the only exception to the rule I can think of for about 5 years. I have friends who say "oh, well if not for injury, RB X or WR Y but I say phoee. Injury is exactly what I am talking about. You have guys who go four years at the collgiate level and never get a scratch. They play one year in the NFL and get injured and you never hear from them again. Its a completely diffrerent level of play and players hold up differently. So, one thing I have built-in in my fantasy drafting is to stay away from rookies unless like anything else, you can get a crazy steal or your willing to take a calculated risk, like Cam Newton was last year. And, BTW I drafted A.J. Green way high last year, but mainly because he was a calculated risk as a WR3 where there was absolutely nobody available (like round 9) unless you consider Jerome Simpson a better 9th round value.

Now a second thing that is kind of ingrained in my fantasy drafting is the three year rule. This is something my son Edward convinced me of and I tend to go with it. It goes something like this ... barring a trade somewhere, a rookie, in his third year with his original team will probably have a career year. Its kinda like the sophmore jinx in baseball. I am sure there is no scientific evidence to support it but I believe in it. So, one thing I will be looking for later when analyzing the position players is which players are in their 3rd year.

But, just to button up this already too long post, we will initially concentrate on things from a team level using the two pieces of information (data) I pasted above; the Vegas odds and last year's final results.

The Three Golden Rules Of Engineering Management

I have been managing Software Engineers for a little over 24 years now and I can honestly say that what I have learned can be broken down into three basic rules …

(1) The Engineers Don't Work for Me, I Work for Them.
I remove blockages. My job is not to control the Engineers, it is to get them operating at peak efficiency so I can get my job done on time. My job is to get the Engineers the tools and knowledge necessary for them to excel. My job is to hold all Engineers accountable for the work scope estimates they give for various use-cases. Once an Engineer, the Product Manager and myself agree a use-case is well understood and an estimate can be given (and this means all reasonable questions posed by the Engineer who will do the work [an important distinction] and myself have been answered by Product [which itself is a bone of contention between Product and Engineering]) I need to make sure said estimate is reasonable (some times Engineers need to be saved from themselves, they often think more highly of themselves then they should) and then met.

(2) A Happy Engineer is a Productive Engineer.
Engineers are typically bright, conscientious people. They want to feel like they accomplished something great today and got paid reasonably well for it. They tend to realize a stable work environment holds value and so they are often reasonable regarding the trade-offs between stability, interesting work and compensation, but rule number one when managing Engineers is you can't bore a good Engineer to death and expect them to be productive. Good Engineers want a real challenge every day. Challenges which result in software people actually use is really reward enough for a good Engineer.

(3) Happy Engineers like Exact Specifications and Dislike Micromanagement.
The establishment of a periodic release cycle requires a user-story to use-case interaction between the Engineering and Product functions within an organization. Additionally, the Engineers need to provide work scope estimates at the use-case level, no lower. Further division into hourly tasks and the attempt to manage at that microscopic a level with good Engineers is counter-productive. The user-story to use-case process which must be done in a collaborative manner between Product and Development is always contentious. The initial set of use-cases will often prove inadequate regardless of how many hours the two sides spend in collaboration. At some point they will need to give in to each other and "do the best they can" for a first release. The establishment early on in the project of a document trail which chronicles what Product thought it was asking for (and associated dates), coupled with an Engineering log which describes what Engineering thought it was being asked to do (also including associated dates), often aids in post-mortem analysis.

UINavigationController - A Simple Example

Recently I decided to use the UINavigationController class so I could push and pop sub views for easy navigation. As usual, all I wanted was a simple clean example to help me get started and of course all the examples I found were either so simple they were useless or they were trying to do much more than the simple example I wanted, as always, here is what I came up with (with base code being copied from some simple example).

This example shows how to navigate between 3 sub views. It includes forward and backward buttons where appropriate.

First the .h file ...

#import <UIKit/UIKit.h>

@interface ViewController : UIViewController
{
    UINavigationController * navController;
    UIBarButtonItem *nextButtonOne;
    UIBarButtonItem *nextButtonTwo;
    UIBarButtonItem *prevButtonTwo;
    UIBarButtonItem *prevButtonThree;
    UIViewController *one;
    UIViewController *two;
    UIViewController *three;
    int currentView;
}

@property (nonatomic, retain)UINavigationController * navController;
@property (nonatomic, retain)UIBarButtonItem *nextButtonOne;
@property (nonatomic, retain)UIBarButtonItem *nextButtonTwo;
@property (nonatomic, retain)UIBarButtonItem *prevButtonTwo;
@property (nonatomic, retain)UIBarButtonItem *prevButtonThree;
@property (nonatomic, retain)UIViewController *one;
@property (nonatomic, retain)UIViewController *two;
@property (nonatomic, retain)UIViewController *three;
@property (atomic, assign)int currentView;

- (IBAction)handleNextButton:(id)sender;
- (IBAction)handlePrevButton:(id)sender;

@end

And next, the .m file ...

#import "ViewController.h"

@implementation ViewController

@synthesize navController, nextButtonOne, nextButtonTwo, currentView;
@synthesize prevButtonTwo, prevButtonThree, one, two, three;

- (void)didReceiveMemoryWarning
{
    [super didReceiveMemoryWarning];
    // Release any cached data, images, etc that aren't in use.
}

#pragma mark - View lifecycle

- (IBAction)handleNextButton:(id)sender {
    if (currentView == 0){
        [navController pushViewController:two animated:YES];
    }else{
        [navController pushViewController:three animated:YES];
    }
    currentView++;
}

- (IBAction)handlePrevButton:(id)sender {
    [navController popViewControllerAnimated:YES];
    currentView--;
}

- (void)viewDidLoad {
    [super viewDidLoad];
    
    one = [[UIViewController alloc] init];

    [one.view setBackgroundColor:[UIColor yellowColor]];
    [one setTitle:@"One"];
    
    nextButtonOne = [[UIBarButtonItem alloc]
                  initWithTitle:@"Two"
                  style:UIBarButtonItemStyleBordered
                  target:self
                  action:@selector(handleNextButton:)];
    
    [[one navigationItem] setRightBarButtonItem:nextButtonOne];

    
    navController = [[UINavigationController alloc] initWithRootViewController:one];
    // here 's the key to the whole thing: we're adding the navController's view to the 
    // self.view, NOT the one.view! So one would be the home page of the app (or something)
    [self.view addSubview:navController.view];
    
    two = [[UIViewController alloc] init];
    [two.view setBackgroundColor:[UIColor blueColor]];
    [two setTitle:@"Two"];
    
    nextButtonTwo = [[UIBarButtonItem alloc]
                     initWithTitle:@"3"
                     style:UIBarButtonItemStyleBordered
                     target:self
                     action:@selector(handleNextButton:)];
    
    prevButtonTwo = [[UIBarButtonItem alloc]
                     initWithTitle:@"1"
                     style:UIBarButtonItemStyleBordered
                     target:self
                     action:@selector(handlePrevButton:)];
    
    [[two navigationItem] setLeftBarButtonItem:prevButtonTwo];
    [[two navigationItem] setRightBarButtonItem:nextButtonTwo];
    
    [navController pushViewController:two animated:YES];
    
    three = [[UIViewController alloc] init];
    [three.view setBackgroundColor:[UIColor redColor]];
    [three setTitle:@"Three"];
    
    prevButtonThree = [[UIBarButtonItem alloc]
                     initWithTitle:@"2"
                     style:UIBarButtonItemStyleBordered
                     target:self
                     action:@selector(handlePrevButton:)];
        
    [[three navigationItem] setLeftBarButtonItem:prevButtonThree];
    [navController pushViewController:three animated:YES];
        
    currentView = 2;
}


- (void)viewDidUnload
{
    [super viewDidUnload];
    // Release any retained subviews of the main view.
    // e.g. self.myOutlet = nil;
}

- (void)viewWillAppear:(BOOL)animated
{
    [super viewWillAppear:animated];
}

- (void)viewDidAppear:(BOOL)animated
{
    [super viewDidAppear:animated];
}

- (void)viewWillDisappear:(BOOL)animated
{
 [super viewWillDisappear:animated];
}

- (void)viewDidDisappear:(BOOL)animated
{
 [super viewDidDisappear:animated];
}

- (BOOL)shouldAutorotateToInterfaceOrientation:(UIInterfaceOrientation)interfaceOrientation
{
    // Return YES for supported orientations
    if ([[UIDevice currentDevice] userInterfaceIdiom] == UIUserInterfaceIdiomPhone) {
        return (interfaceOrientation != UIInterfaceOrientationPortraitUpsideDown);
    } else {
        return YES;
    }
}

@end

Its that simple :-)

Websockets Revisited - Part 2

This is part 2 of a 3 part post covering my recent experiences with websockets. In this section I will just post the source code and show you how to test it out. In the next section (if your interested) I will go over the changes I made to websocket.py and client.html to get it working with both Safari and Chrome on my MacBook Pro.

Copy and paste the following 3 files into the same directory on your existing websever. All you need to change is one line in the client.html file. Line 41 should look like this ...

ws = new WebSocket("ws://174.143.214.70:9876/");

Just change the IP address to point to your webserver IP address. Once that is done you run ...

python example.py

Then you point your browser to client.html and type something in the message input field and press the button next to it and if you see a response in the browser then it is working. I did not update the demo to show a timer tic or to broadcast the message to all connected clients. That is left as an exercise for the reader :-)

So here is client.html ...

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>Test</title>
        <script type="application/javascript">
            var ws;

            var file, fr, new_str;


        function receivedText() {
           result = fr.result;
           new_str = "";
           for (n = 0; n < result.length; ++n) {
              aByte = result.charCodeAt(n);
              new_str += String.fromCharCode(aByte);
           }
//           alert(new_str);
handleCommand();
        }

function handleCommand(){
                    //da = e.data.split(":");
                    da = new_str.split(":");


                    cmd = da[0];
                    data = da[1];
//                    servermsg.innerHTML = servermsg.innerHTML + "<br>Recieved data: " + e.data;
                    if (cmd == "tick"){
                      tic.innerHTML = data;
                    }else{
                      servermsg.innerHTML = servermsg.innerHTML + "<br>Rcvd: " + data;
                    }
}


            function init() {
                var servermsg = document.getElementById("servermsg");

                ws = new WebSocket("ws://174.143.214.70:9876/");
                ws.onopen = function(){
                    servermsg.innerHTML = servermsg.innerHTML + "<br>Server connected";
                };
                ws.onmessage = function(e){

x = e.data;
retcode = (x instanceof Blob);

if (retcode){
// this works for chrome who returns a blob object from the websocket
            new_str = "";
            file = e.data;
            fr = new FileReader();
            fr.onload = receivedText;
            fr.readAsText(file);
}else{
    new_str = e.data;
    handleCommand();
}

                };
                ws.onclose = function(){
                    console.log("Server disconnected");
                    servermsg.innerHTML = servermsg.innerHTML + "<br>Disconnected";
                };
            }
            function postmsg(){
                var text = document.getElementById("message").value;
                text = "msg:" + text;
                ws.send(text);
                servermsg.innerHTML = servermsg.innerHTML + "<br>Sent: " + text;
                return false;
            }
        </script>
    </head>
    <body onload="init();">
      <span id="tic">0</span><br/>
      <input type="text" name="message" value="" id="message">
      <button onClick="postmsg();">Send</button>
      <div id="servermsg" style="overflow:auto; height:400px; width:400px; border:1px solid black"><h1>Message log:</h1></div>
    </body>
</html>

Yes, I am a terrible hack; I know :-(

Next, we have the example.py file ...

import websocket
import time


#bind = ('127.0.0.1', 8888)
class MyWrap(websocket.WebSocketServer):
    pass

    def new_client(self):
        while 1:
            bufs_list, closed_string = self.recv_frames()
            print "Received --->", bufs_list
            ret_code = self.send_frames(bufs=bufs_list)
            print "ret code from send is ", ret_code

    def poll(self):
        #print "Inside Poll"
        pass
       
if __name__ == "__main__":
    server = MyWrap(verbose=True, listen_host='', listen_port=9876)
    server.start_server()
    while 1:
        time.sleep(1)

And finally websocket.py ...

#!/usr/bin/env python

'''
Python WebSocket library with support for "wss://" encryption.
Copyright 2011 Joel Martin
Licensed under LGPL version 3 (see docs/LICENSE.LGPL-3)

Supports following protocol versions:
- http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-75
- http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-76
- http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-10

You can make a cert/key with openssl using:
openssl req -new -x509 -days 365 -nodes -out self.pem -keyout self.pem
as taken from http://docs.python.org/dev/library/ssl.html#certificates

'''

import os, sys, time, errno, signal, socket, traceback, select
import array, struct
from cgi import parse_qsl
from base64 import b64encode, b64decode

# Imports that vary by python version

# python 3.0 differences
if sys.hexversion > 0x3000000:
    b2s = lambda buf: buf.decode('latin_1')
    s2b = lambda s: s.encode('latin_1')
    s2a = lambda s: s
else:
    b2s = lambda buf: buf # No-op
    s2b = lambda s: s # No-op
    s2a = lambda s: [ord(c) for c in s]
try: from io import StringIO
except: from cStringIO import StringIO
try: from http.server import SimpleHTTPRequestHandler
except: from SimpleHTTPServer import SimpleHTTPRequestHandler
try: from urllib.parse import urlsplit
except: from urlparse import urlsplit

# python 2.6 differences
try: from hashlib import md5, sha1
except: from md5 import md5; from sha import sha as sha1

# python 2.5 differences
try:
    from struct import pack, unpack_from
except:
    from struct import pack
    def unpack_from(fmt, buf, offset=0):
        slice = buffer(buf, offset, struct.calcsize(fmt))
        return struct.unpack(fmt, slice)

# Degraded functionality if these imports are missing
for mod, sup in [('numpy', 'HyBi protocol'), ('ssl', 'TLS/SSL/wss'),
        ('multiprocessing', 'Multi-Processing'),
        ('resource', 'daemonizing')]:
    try:
        globals()[mod] = __import__(mod)
    except ImportError:
        globals()[mod] = None
        print("WARNING: no '%s' module, %s is slower or disabled" % (
            mod, sup))
if multiprocessing and sys.platform == 'win32':
    # make sockets pickle-able/inheritable
    import multiprocessing.reduction


class WebSocketServer(object):
    """
WebSockets server class.
Must be sub-classed with new_client method definition.
"""

    buffer_size = 65536

    server_handshake_hixie = """HTTP/1.1 101 Web Socket Protocol Handshake\r
Upgrade: WebSocket\r
Connection: Upgrade\r
%sWebSocket-Origin: %s\r
%sWebSocket-Location: %s://%s%s\r
"""

    server_handshake_hybi = """HTTP/1.1 101 Switching Protocols\r
Upgrade: websocket\r
Connection: Upgrade\r
Sec-WebSocket-Accept: %s\r
"""

    GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"

    policy_response = """<cross-domain-policy><allow-access-from domain="*" to-ports="*" /></cross-domain-policy>\n"""

    class EClose(Exception):
        pass

    def __init__(self, listen_host='', listen_port=None, source_is_ipv6=False,
            verbose=False, cert='', key='', ssl_only=None,
            daemon=False, record='', web='',
            run_once=False, timeout=0):

        # settings
        self.verbose = verbose
        self.listen_host = listen_host
        self.listen_port = listen_port
        self.ssl_only = ssl_only
        self.daemon = daemon
        self.run_once = run_once
        self.timeout = timeout

        self.launch_time = time.time()
        self.ws_connection = False
        self.handler_id = 1

        # Make paths settings absolute
        self.cert = os.path.abspath(cert)
        self.key = self.web = self.record = ''
        if key:
            self.key = os.path.abspath(key)
        if web:
            self.web = os.path.abspath(web)
        if record:
            self.record = os.path.abspath(record)

        if self.web:
            os.chdir(self.web)

        # Sanity checks
        if not ssl and self.ssl_only:
            raise Exception("No 'ssl' module and SSL-only specified")
        if self.daemon and not resource:
            raise Exception("Module 'resource' required to daemonize")

        # Show configuration
        print("WebSocket server settings:")
        print(" - Listen on %s:%s" % (
                self.listen_host, self.listen_port))
        print(" - Flash security policy server")
        if self.web:
            print(" - Web server. Web root: %s" % self.web)
        if ssl:
            if os.path.exists(self.cert):
                print(" - SSL/TLS support")
                if self.ssl_only:
                    print(" - Deny non-SSL/TLS connections")
            else:
                print(" - No SSL/TLS support (no cert file)")
        else:
            print(" - No SSL/TLS support (no 'ssl' module)")
        if self.daemon:
            print(" - Backgrounding (daemon)")
        if self.record:
            print(" - Recording to '%s.*'" % self.record)

    #
    # WebSocketServer static methods
    #

    @staticmethod
    def socket(host, port=None, connect=False, prefer_ipv6=False):
        """ Resolve a host (and optional port) to an IPv4 or IPv6
address. Create a socket. Bind to it if listen is set,
otherwise connect to it. Return the socket.
"""
        flags = 0
        if host == '':
            host = None
        if connect and not port:
            raise Exception("Connect mode requires a port")
        if not connect:
            flags = flags | socket.AI_PASSIVE
        addrs = socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM,
                socket.IPPROTO_TCP, flags)
        if not addrs:
            raise Exception("Could resolve host '%s'" % host)
        addrs.sort(key=lambda x: x[0])
        if prefer_ipv6:
            addrs.reverse()
        sock = socket.socket(addrs[0][0], addrs[0][1])
        if connect:
            sock.connect(addrs[0][4])
        else:
            sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
            sock.bind(addrs[0][4])
            sock.listen(100)
        return sock

    @staticmethod
    def daemonize(keepfd=None, chdir='/'):
        os.umask(0)
        if chdir:
            os.chdir(chdir)
        else:
            os.chdir('/')
        os.setgid(os.getgid()) # relinquish elevations
        os.setuid(os.getuid()) # relinquish elevations

        # Double fork to daemonize
        if os.fork() > 0: os._exit(0) # Parent exits
        os.setsid() # Obtain new process group
        if os.fork() > 0: os._exit(0) # Parent exits

        # Signal handling
        def terminate(a,b): os._exit(0)
        signal.signal(signal.SIGTERM, terminate)
        signal.signal(signal.SIGINT, signal.SIG_IGN)

        # Close open files
        maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
        if maxfd == resource.RLIM_INFINITY: maxfd = 256
        for fd in reversed(range(maxfd)):
            try:
                if fd != keepfd:
                    os.close(fd)
            except OSError:
                _, exc, _ = sys.exc_info()
                if exc.errno != errno.EBADF: raise

        # Redirect I/O to /dev/null
        os.dup2(os.open(os.devnull, os.O_RDWR), sys.stdin.fileno())
        os.dup2(os.open(os.devnull, os.O_RDWR), sys.stdout.fileno())
        os.dup2(os.open(os.devnull, os.O_RDWR), sys.stderr.fileno())

    @staticmethod
    def unmask(buf, f):
        pstart = f['hlen'] + 4
        pend = pstart + f['length']
        if numpy:
            b = c = s2b('')
            if f['length'] >= 4:
                mask = numpy.frombuffer(buf, dtype=numpy.dtype('<u4'),
                        offset=f['hlen'], count=1)
                data = numpy.frombuffer(buf, dtype=numpy.dtype('<u4'),
                        offset=pstart, count=int(f['length'] / 4))
                #b = numpy.bitwise_xor(data, mask).data
                b = numpy.bitwise_xor(data, mask).tostring()

            if f['length'] % 4:
                #print("Partial unmask")
                mask = numpy.frombuffer(buf, dtype=numpy.dtype('B'),
                        offset=f['hlen'], count=(f['length'] % 4))
                data = numpy.frombuffer(buf, dtype=numpy.dtype('B'),
                        offset=pend - (f['length'] % 4),
                        count=(f['length'] % 4))
                c = numpy.bitwise_xor(data, mask).tostring()
            return b + c
        else:
            # Slower fallback
            data = array.array('B')
            mask = s2a(f['mask'])
            data.fromstring(buf[pstart:pend])
            for i in range(len(data)):
                data[i] ^= mask[i % 4]
            return data.tostring()

    @staticmethod
    def encode_hybi(buf, opcode, base64=False):
        """ Encode a HyBi style WebSocket frame.
Optional opcode:
0x0 - continuation
0x1 - text frame (base64 encode buf)
0x2 - binary frame (use raw buf)
0x8 - connection close
0x9 - ping
0xA - pong
"""
        if base64:
            buf = b64encode(buf)

        b1 = 0x80 | (opcode & 0x0f) # FIN + opcode
        payload_len = len(buf)
        if payload_len <= 125:
            header = pack('>BB', b1, payload_len)
        elif payload_len > 125 and payload_len < 65536:
            header = pack('>BBH', b1, 126, payload_len)
        elif payload_len >= 65536:
            header = pack('>BBQ', b1, 127, payload_len)

        #print("Encoded: %s" % repr(header + buf))

        return header + buf, len(header), 0

    @staticmethod
    def decode_hybi(buf, base64=False):
        """ Decode HyBi style WebSocket packets.
Returns:
{'fin' : 0_or_1,
'opcode' : number,
'mask' : 32_bit_number,
'hlen' : header_bytes_number,
'length' : payload_bytes_number,
'payload' : decoded_buffer,
'left' : bytes_left_number,
'close_code' : number,
'close_reason' : string}
"""

        f = {'fin' : 0,
             'opcode' : 0,
             'mask' : 0,
             'hlen' : 2,
             'length' : 0,
             'payload' : None,
             'left' : 0,
             'close_code' : None,
             'close_reason' : None}

        blen = len(buf)
        f['left'] = blen

        if blen < f['hlen']:
            return f # Incomplete frame header

        b1, b2 = unpack_from(">BB", buf)
        f['opcode'] = b1 & 0x0f
        f['fin'] = (b1 & 0x80) >> 7
        has_mask = (b2 & 0x80) >> 7

        f['length'] = b2 & 0x7f

        if f['length'] == 126:
            f['hlen'] = 4
            if blen < f['hlen']:
                return f # Incomplete frame header
            (f['length'],) = unpack_from('>xxH', buf)
        elif f['length'] == 127:
            f['hlen'] = 10
            if blen < f['hlen']:
                return f # Incomplete frame header
            (f['length'],) = unpack_from('>xxQ', buf)

        full_len = f['hlen'] + has_mask * 4 + f['length']

        if blen < full_len: # Incomplete frame
            return f # Incomplete frame header

        # Number of bytes that are part of the next frame(s)
        f['left'] = blen - full_len

        # Process 1 frame
        if has_mask:
            # unmask payload
            f['mask'] = buf[f['hlen']:f['hlen']+4]
            f['payload'] = WebSocketServer.unmask(buf, f)
        else:
            print("Unmasked frame: %s" % repr(buf))
            f['payload'] = buf[(f['hlen'] + has_mask * 4):full_len]

        if base64 and f['opcode'] in [1, 2]:
            try:
                f['payload'] = b64decode(f['payload'])
            except:
                print("Exception while b64decoding buffer: %s" %
                        repr(buf))
                raise

        if f['opcode'] == 0x08:
            if f['length'] >= 2:
                f['close_code'] = unpack_from(">H", f['payload'])
            if f['length'] > 3:
                f['close_reason'] = f['payload'][2:]

        return f

    @staticmethod
    def OLD_encode_hixie(buf):
        return s2b("\x00" + b2s(b64encode(buf)) + "\xff"), 1, 1

    @staticmethod
    def encode_hixie(buf):
        return "\x00" + buf + "\xff", 1, 1

    @staticmethod
    def decode_hixie(buf):
        end = buf.find(s2b('\xff'))

#        return {'payload': b64decode(buf[1:end]),
#                'hlen': 1,
#                'length': end - 1,
#                'left': len(buf) - (end + 1)}

        return {'payload': buf[1:end],
                'hlen': 1,
                'length': end - 1,
                'left': len(buf) - (end + 1)}

    @staticmethod
    def gen_md5(keys):
        """ Generate hash value for WebSockets hixie-76. """
        key1 = keys['Sec-WebSocket-Key1']
        key2 = keys['Sec-WebSocket-Key2']
        key3 = keys['key3']
        spaces1 = key1.count(" ")
        spaces2 = key2.count(" ")
        num1 = int("".join([c for c in key1 if c.isdigit()])) / spaces1
        num2 = int("".join([c for c in key2 if c.isdigit()])) / spaces2

        return b2s(md5(pack('>II8s',
            int(num1), int(num2), key3)).digest())

    #
    # WebSocketServer logging/output functions
    #

    def traffic(self, token="."):
        """ Show traffic flow in verbose mode. """
        if self.verbose and not self.daemon:
            sys.stdout.write(token)
            sys.stdout.flush()

    def msg(self, msg):
        """ Output message with handler_id prefix. """
        if not self.daemon:
            print("% 3d: %s" % (self.handler_id, msg))

    def vmsg(self, msg):
        """ Same as msg() but only if verbose. """
        if self.verbose:
            self.msg(msg)

    #
    # Main WebSocketServer methods
    #
    def send_frames(self, bufs=None):
        """ Encode and send WebSocket frames. Any frames already
queued will be sent first. If buf is not set then only queued
frames will be sent. Returns the number of pending frames that
could not be fully sent. If returned pending frames is greater
than 0, then the caller should call again when the socket is
ready. """

        tdelta = int(time.time()*1000) - self.start_time

        if bufs:
            for buf in bufs:
                print "XXXXXX VERSION ", self.version
                if self.version.startswith("hybi"):
                    if self.base64:
                        encbuf, lenhead, lentail = self.encode_hybi(
                                buf, opcode=1, base64=True)
                    else:
                        encbuf, lenhead, lentail = self.encode_hybi(
                                buf, opcode=2, base64=False)

                else:
                    encbuf, lenhead, lentail = self.encode_hixie(buf)

                if self.rec:
                    self.rec.write("%s,\n" %
                            repr("{%s{" % tdelta
                                + encbuf[lenhead:-lentail]))

                self.send_parts.append(encbuf)

        while self.send_parts:
            # Send pending frames
            buf = self.send_parts.pop(0)

            print "YYYYYYY Sending this to client ", buf
            for l in buf:
                print "%2x" % (ord(l))

            sent = self.client.send(buf)

            if sent == len(buf):
                self.traffic("<")
            else:
                self.traffic("<.")
                self.send_parts.insert(0, buf[sent:])
                break

        return len(self.send_parts)

    def recv_frames(self):
        """ Receive and decode WebSocket frames.

Returns:
(bufs_list, closed_string)
"""

        closed = False
        bufs = []
        tdelta = int(time.time()*1000) - self.start_time

        buf = self.client.recv(self.buffer_size)
        if len(buf) == 0:
            closed = "Client closed abruptly"
            return bufs, closed

        if self.recv_part:
            # Add partially received frames to current read buffer
            buf = self.recv_part + buf
            self.recv_part = None

        while buf:
            if self.version.startswith("hybi"):

                frame = self.decode_hybi(buf, base64=self.base64)
                #print("Received buf: %s, frame: %s" % (repr(buf), frame))

                if frame['payload'] == None:
                    # Incomplete/partial frame
                    self.traffic("}.")
                    if frame['left'] > 0:
                        self.recv_part = buf[-frame['left']:]
                    break
                else:
                    if frame['opcode'] == 0x8: # connection close
                        closed = "Client closed, reason: %s - %s" % (
                                frame['close_code'],
                                frame['close_reason'])
                        break

            else:
                if buf[0:2] == s2b('\xff\x00'):
                    closed = "Client sent orderly close frame"
                    break

                elif buf[0:2] == s2b('\x00\xff'):
                    buf = buf[2:]
                    continue # No-op

                elif buf.count(s2b('\xff')) == 0:
                    # Partial frame
                    self.traffic("}.")
                    self.recv_part = buf
                    break

                frame = self.decode_hixie(buf)

            self.traffic("}")

            if self.rec:
                start = frame['hlen']
                end = frame['hlen'] + frame['length']
                self.rec.write("%s,\n" %
                        repr("}%s}" % tdelta + buf[start:end]))


            bufs.append(frame['payload'])

            if frame['left']:
                buf = buf[-frame['left']:]
            else:
                buf = ''

        return bufs, closed

    def send_close(self, code=None, reason=''):
        """ Send a WebSocket orderly close frame. """

        if self.version.startswith("hybi"):
            msg = s2b('')
            if code != None:
                msg = pack(">H%ds" % (len(reason)), code)

            buf, h, t = self.encode_hybi(msg, opcode=0x08, base64=False)
            self.client.send(buf)

        elif self.version == "hixie-76":
            buf = s2b('\xff\x00')
            self.client.send(buf)

        # No orderly close for 75

    def do_handshake(self, sock, address):
        """
do_handshake does the following:
- Peek at the first few bytes from the socket.
- If the connection is Flash policy request then answer it,
close the socket and return.
- If the connection is an HTTPS/SSL/TLS connection then SSL
wrap the socket.
- Read from the (possibly wrapped) socket.
- If we have received a HTTP GET request and the webserver
functionality is enabled, answer it, close the socket and
return.
- Assume we have a WebSockets connection, parse the client
handshake data.
- Send a WebSockets handshake server response.
- Return the socket for this WebSocket client.
"""

        stype = ""

        ready = select.select([sock], [], [], 3)[0]
        if not ready:
            raise self.EClose("ignoring socket not ready")
        # Peek, but do not read the data so that we have a opportunity
        # to SSL wrap the socket first
        handshake = sock.recv(1024, socket.MSG_PEEK)
        #self.msg("Handshake [%s]" % handshake)

        if handshake == "":
            raise self.EClose("ignoring empty handshake")

        elif handshake.startswith(s2b("<policy-file-request/>")):
            # Answer Flash policy request
            handshake = sock.recv(1024)
            sock.send(s2b(self.policy_response))
            raise self.EClose("Sending flash policy response")

        elif handshake[0] in ("\x16", "\x80", 22, 128):
            # SSL wrap the connection
            if not ssl:
                raise self.EClose("SSL connection but no 'ssl' module")
            if not os.path.exists(self.cert):
                raise self.EClose("SSL connection but '%s' not found"
                                  % self.cert)
            retsock = None
            try:
                retsock = ssl.wrap_socket(
                        sock,
                        server_side=True,
                        certfile=self.cert,
                        keyfile=self.key)
            except ssl.SSLError:
                _, x, _ = sys.exc_info()
                if x.args[0] == ssl.SSL_ERROR_EOF:
                    raise self.EClose(x.args[1])
                else:
                    raise

            scheme = "wss"
            stype = "SSL/TLS (wss://)"

        elif self.ssl_only:
            raise self.EClose("non-SSL connection received but disallowed")

        else:
            retsock = sock
            scheme = "ws"
            stype = "Plain non-SSL (ws://)"

        wsh = WSRequestHandler(retsock, address, not self.web)
        if wsh.last_code == 101:
            # Continue on to handle WebSocket upgrade
            pass
        elif wsh.last_code == 405:
            raise self.EClose("Normal web request received but disallowed")
        elif wsh.last_code < 200 or wsh.last_code >= 300:
            raise self.EClose(wsh.last_message)
        elif self.verbose:
            raise self.EClose(wsh.last_message)
        else:
            raise self.EClose("")

        h = self.headers = wsh.headers
        path = self.path = wsh.path

        prot = 'WebSocket-Protocol'
        protocols = h.get('Sec-'+prot, h.get(prot, '')).split(',')

        ver = h.get('Sec-WebSocket-Version')
        if ver:
            # HyBi/IETF version of the protocol

            # HyBi-07 report version 7
            # HyBi-08 - HyBi-12 report version 8
            # HyBi-13 reports version 13
            if ver in ['7', '8', '13']:
                self.version = "hybi-%02d" % int(ver)
            else:
                raise self.EClose('Unsupported protocol version %s' % ver)

            key = h['Sec-WebSocket-Key']

            # Choose binary if client supports it
            if 'binary' in protocols:
                self.base64 = False
            elif 'base64' in protocols:
                self.base64 = True
            else:
                #raise self.EClose("Client must support 'binary' or 'base64' protocol")
                print "XXX SHIT!!! client does not support binary or base64 WTF???"
                self.base64 = False

            # Generate the hash value for the accept header
            accept = b64encode(sha1(s2b(key + self.GUID)).digest())

            response = self.server_handshake_hybi % b2s(accept)
            if self.base64:
                response += "Sec-WebSocket-Protocol: base64\r\n"
            else:
                response += "Sec-WebSocket-Protocol: binary\r\n"
            response += "\r\n"

        else:
            # Hixie version of the protocol (75 or 76)

            if h.get('key3'):
                trailer = self.gen_md5(h)
                pre = "Sec-"
                self.version = "hixie-76"
            else:
                trailer = ""
                pre = ""
                self.version = "hixie-75"

            # We only support base64 in Hixie era
            self.base64 = True

            response = self.server_handshake_hixie % (pre,
                    h['Origin'], pre, scheme, h['Host'], path)

            if 'base64' in protocols:
                response += "%sWebSocket-Protocol: base64\r\n" % pre
            else:
                self.msg("Warning: client does not report 'base64' protocol support")
            response += "\r\n" + trailer

        self.msg("%s: %s WebSocket connection" % (address[0], stype))
        self.msg("%s: Version %s, base64: '%s'" % (address[0],
            self.version, self.base64))
        if self.path != '/':
            self.msg("%s: Path: '%s'" % (address[0], self.path))


        # Send server WebSockets handshake response
        #self.msg("sending response [%s]" % response)
        retsock.send(s2b(response))

        # Return the WebSockets socket which may be SSL wrapped
        return retsock


    #
    # Events that can/should be overridden in sub-classes
    #
    def started(self):
        """ Called after WebSockets startup """
        self.vmsg("WebSockets server started")

    def poll(self):
        """ Run periodically while waiting for connections. """
        #self.vmsg("Running poll()")
        pass

    def fallback_SIGCHLD(self, sig, stack):
        # Reap zombies when using os.fork() (python 2.4)
        self.vmsg("Got SIGCHLD, reaping zombies")
        try:
            result = os.waitpid(-1, os.WNOHANG)
            while result[0]:
                self.vmsg("Reaped child process %s" % result[0])
                result = os.waitpid(-1, os.WNOHANG)
        except (OSError):
            pass

    def do_SIGINT(self, sig, stack):
        self.msg("Got SIGINT, exiting")
        sys.exit(0)

    def top_new_client(self, startsock, address):
        """ Do something with a WebSockets client connection. """
        # Initialize per client settings
        self.send_parts = []
        self.recv_part = None
        self.base64 = False
        self.rec = None
        self.start_time = int(time.time()*1000)

        # handler process
        try:
            try:
                self.client = self.do_handshake(startsock, address)

                if self.record:
                    # Record raw frame data as JavaScript array
                    fname = "%s.%s" % (self.record,
                                        self.handler_id)
                    self.msg("opening record file: %s" % fname)
                    self.rec = open(fname, 'w+')
                    self.rec.write("var VNC_frame_data = [\n")

                self.ws_connection = True
                self.new_client()
            except self.EClose:
                _, exc, _ = sys.exc_info()
                # Connection was not a WebSockets connection
                if exc.args[0]:
                    self.msg("%s: %s" % (address[0], exc.args[0]))
            except Exception:
                _, exc, _ = sys.exc_info()
                self.msg("handler exception: %s" % str(exc))
                if self.verbose:
                    self.msg(traceback.format_exc())
        finally:
            if self.rec:
                self.rec.write("'EOF']\n")
                self.rec.close()

            if self.client and self.client != startsock:
                self.client.close()

    def new_client(self):
        """ Do something with a WebSockets client connection. """
        raise("WebSocketServer.new_client() must be overloaded")

    def start_server(self):
        """
Daemonize if requested. Listen for for connections. Run
do_handshake() method for each connection. If the connection
is a WebSockets client then call new_client() method (which must
be overridden) for each new client connection.
"""
        lsock = self.socket(self.listen_host, self.listen_port)

        if self.daemon:
            self.daemonize(keepfd=lsock.fileno(), chdir=self.web)

        self.started() # Some things need to happen after daemonizing

        # Allow override of SIGINT
        signal.signal(signal.SIGINT, self.do_SIGINT)
        if not multiprocessing:
            # os.fork() (python 2.4) child reaper
            signal.signal(signal.SIGCHLD, self.fallback_SIGCHLD)

        while True:
            try:
                try:
                    self.client = None
                    startsock = None
                    pid = err = 0

                    time_elapsed = time.time() - self.launch_time
                    if self.timeout and time_elapsed > self.timeout:
                        self.msg('listener exit due to --timeout %s'
                                % self.timeout)
                        break

                    try:
                        self.poll()

                        ready = select.select([lsock], [], [], 1)[0]
                        if lsock in ready:
                            startsock, address = lsock.accept()
                        else:
                            continue
                    except Exception:
                        _, exc, _ = sys.exc_info()
                        if hasattr(exc, 'errno'):
                            err = exc.errno
                        elif hasattr(exc, 'args'):
                            err = exc.args[0]
                        else:
                            err = exc[0]
                        if err == errno.EINTR:
                            self.vmsg("Ignoring interrupted syscall")
                            continue
                        else:
                            raise

                    if self.run_once:
                        # Run in same process if run_once
                        self.top_new_client(startsock, address)
                        if self.ws_connection :
                            self.msg('%s: exiting due to --run-once'
                                    % address[0])
                            break
                    elif multiprocessing:
                        self.vmsg('%s: new handler Process' % address[0])
                        p = multiprocessing.Process(
                                target=self.top_new_client,
                                args=(startsock, address))
                        p.start()
                        # child will not return
                    else:
                        # python 2.4
                        self.vmsg('%s: forking handler' % address[0])
                        pid = os.fork()
                        if pid == 0:
                            # child handler process
                            self.top_new_client(startsock, address)
                            break # child process exits

                    # parent process
                    self.handler_id += 1

                except KeyboardInterrupt:
                    _, exc, _ = sys.exc_info()
                    print("In KeyboardInterrupt")
                    pass
                except SystemExit:
                    _, exc, _ = sys.exc_info()
                    print("In SystemExit")
                    break
                except Exception:
                    _, exc, _ = sys.exc_info()
                    self.msg("handler exception: %s" % str(exc))
                    if self.verbose:
                        self.msg(traceback.format_exc())

            finally:
                if startsock:
                    startsock.close()


# HTTP handler with WebSocket upgrade support
class WSRequestHandler(SimpleHTTPRequestHandler):
    def __init__(self, req, addr, only_upgrade=False):
        self.only_upgrade = only_upgrade # only allow upgrades
        SimpleHTTPRequestHandler.__init__(self, req, addr, object())

    def do_GET(self):
        if (self.headers.get('upgrade') and
                self.headers.get('upgrade').lower() == 'websocket'):

            if (self.headers.get('sec-websocket-key1') or
                    self.headers.get('websocket-key1')):
                # For Hixie-76 read out the key hash
                self.headers.__setitem__('key3', self.rfile.read(8))

            # Just indicate that an WebSocket upgrade is needed
            self.last_code = 101
            self.last_message = "101 Switching Protocols"
        elif self.only_upgrade:
            # Normal web request responses are disabled
            self.last_code = 405
            self.last_message = "405 Method Not Allowed"
        else:
            SimpleHTTPRequestHandler.do_GET(self)

    def send_response(self, code, message=None):
        # Save the status code
        self.last_code = code
        SimpleHTTPRequestHandler.send_response(self, code, message)

    def log_message(self, f, *args):
        # Save instead of printing
        self.last_message = f % args

Again, I apologize for the hacky code (it really is a shame as websocket.py is written so well) but I left the affected area's logging in to enunciate the differences. It will also dump out the request so you can see the differences in the headers. In the next post I will incorporate the changes better so they are backward compatibile with the original code.

Anyway, this demo should work with Safari and Chrome on a MacBook Pro. Your mileage may vary.

Websockets Revisited - Part 1

So in my last post I naively stated that websockets were easy. What a difference a few days makes. It turns out I can only find two browsers for my MacBook Pro that currently support websockets and the code on both the client and the server must be different for each. As I mentioned to my friend Scott the other day, I hope this doesn't turn out like Ajax did where you have all sorts of cross browser compatibility issues because so far that is exactly what is happening.

A bit of background first. The websocket spec RFC 6455 (http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-17) covers the websocket protocol. The first version "draft-ietf-hybi-thewebsocketprotocol-00" which is what my version of Safari (5.1.3) supports, framed its data using a 0x00 byte to start the frame and a 0xff byte to end the frame. It had two security keys in the handshake header and the response to these keys was calculated by concatenating the numbers from the first key, and dividing by the number of spaces. This is then repeated for the second key. The two resulting numbers are concatenated with each other, and with the last 8 bytes after the fields. The final result is an MD5 sum of the concatenated string (this is from the websocket wiki page).

Obviously, the server will need to do some work to provide a proper handshake to the browser. Specifically, it must interpret the header and indicate to the client it has done so by providing a return value derived from the two security keys in the header (ugh!).

Now, while the code in my last post indeed works with Safari, it does not work with Chrome (version 17.0.963.56). That's because Chrome uses the newer version of the websocket protocol; "draft-ietf-hybi-thewebsocketprotocol-06". This version was created because of man-in-the-middle attacks and other security vulnerabilities so now we have a completely different, non-backward compatible version of websockets floating around out there. In this second version of the specification, there is not two but one security key and its value is calculated differently. To make matters worse, there is a bit-oriented header and a 4 byte xor mask passed with each frame and the actual frame data is 'masked' (uber ugh!).

Of course, this is just the server side. My experience on the client side is that Safari passes websocket data back to the Javascript application as a string but Chrome passes it back as a blob. Being new to blobs (and HTML5 files) it took me a while to figure out how to deal with the blob data. The thing that concerns me is not the different ways the data is presented to the application (blob vs string) but that it is different at all. Don't these guys who write browsers communicate? This is why I think Javascript tool kits like Dojo and jQuery are safe at least for the near future.

Anyway, as you can see from the above descriptions of header parsing and handshaking, this is a lot more code than I feel like writing just to get a simple websocket demo working, so I started looking for websocket servers. Now, while I can code in Assembly, C, C++, Java, PERL, PHP, Python, Ruby and Javascript (as well as Pascal and Fortran but we won't go there) I would prefer the server to be written in Python since that is what I am most comfortable with lately and specifically, for most of my personal projects (because I always assume my project will become so successful it will need to scale to gazillions of users) I tend to use Tornado (my favorite Python webserver).

It turns out there is probably a websocket server written in any language you like (including erlang :-) and so I started reading up on them. From what I read, the Tornado websocket support is buggy (but I could be wrong) so I kept away from it; for now. I do not like big ass frameworks and many of the servers I looked at were add-ons to existing systems. I wanted a simple piece of code my simple mind could comprehend so heaven forbid, if it didn't work I could possibly debug it (this turned out to be prophetic to say the least) and as a result I settled on "websocket.py".

When I brought up "websocket.py" it did not work with my version of Safari or Chrome. Hell, I even got Safari working with a small piece of code 2 days ago, so I figured I would roll up my shirt sleeves and hack around "websocket.py" and make it work. I will go over the code mods and the new code in the next blog - "Websockets Revisited - Part 2". It concerns me though because I am sure whoever wrote "websocket.py" tested it on something and it worked so my mods have probably broken something that was working, but, I also know that what was not working with my two browsers is now working so I'll try to relay this information to the original author of "websocket.py".

Websockets and Python

I had recently built a push notification framework using AJAX and a FireFox quirk described here (http://ajaxpatterns.org/HTTP_Streaming) with a Tornado webserver to keep the socket open ala long poll, however, this only works on Firefox and is really a kludgy way to do things so I recently bit the bullet and dove into HTML5 Websockets.

Even though Websockets are not supported on Firefox yet (and probably not IE, I don't know and after working with JavaScript over the past 10 years I really don't care about IE) I think moving forward this is the way to go. What I discovered is using Websockets is really really easy. I copied some basic code from StackOverflow (http://stackoverflow.com/questions/4372657/websocket-handshake-problem-using-python-server) modified it to be a little easier on the eyes and included a universal timer tick so all users can see they are in sync in real-time, and viola - we have the following 2 really small code snippets.

To test this app on your environment you will need a separate web server that can serve out the .html page to a browser on the web. I use APACHE but any web server will do including the one built into python which can be run like this ...

sudo python -m SimpleHTTPServer 80

If you want it to handle port 80 (http), or ...

python -m SimpleHTTPServer 8888

If you want it to run on a simple user port.

Once you have that working so you can point a browser to your client.html page and run it, you can start your python server (the code below) like this ...

python server.py

This assumes you name the client file 'client.html' and the server file 'server.py'. Note this is not an optimal solution as I am not sure how well this approach will scale (python threads are not real fast). I usually use Tornado for my personal projects that have to scale but I read that Tornado is not working well with Websockets yet. If anyone knows otherwise, please let me know.

Anyway, assuming all is well, this will show a page with a universal timer tick which should be in sync across all connected browsers/users and a message area where any messages sent from anyone currently connected are displayed.

Don't forget to modify your code to use your IP address. The IP address in the code points to a mini dev server I have which may or may not be active when you try out the code. Also, I have only tested it on Safari. It may or may not work on other browsers.

Now that I have this code working I can use it to build my HTML5 2D Game Engine framework which I will release in a future post.

Now on to the actual code ...


First, the client code (JavaScript in HTML) ...

<!DOCTYPE html>
<html lang="en">
 <head>
     <title>Test</title>
     <script type="application/javascript">
         var ws;

         function init() {
             var servermsg = document.getElementById("servermsg");

             ws = new WebSocket("ws://174.143.214.70:9876/");
             ws.onopen = function(){
                 servermsg.innerHTML = servermsg.innerHTML + "<br>Server connected";
             };
             ws.onmessage = function(e){
                 da = e.data.split(":");
                 cmd = da[0];
                 data = da[1];
//                    servermsg.innerHTML = servermsg.innerHTML + "<br>Recieved data: " + e.data;
                 if (cmd == "tick"){
                   tic.innerHTML = data;
                 }else{
                   servermsg.innerHTML = servermsg.innerHTML + "<br>" + data;
                 }
             };
             ws.onclose = function(){
                 console.log("Server disconnected");
                 servermsg.innerHTML = servermsg.innerHTML + "<br>Disconnected";
             };
         }
         function postmsg(){
             var text = document.getElementById("message").value;
             ws.send(text);
             servermsg.innerHTML = servermsg.innerHTML + "<br>Sent: " + text;
             return false;
         }
     </script>
 </head>
 <body onload="init();">
   <span id="tic">0</span><br/>
   <input type="text" name="message" value="" id="message">
   <button onClick="postmsg();">Send</button>
   <div id="servermsg" style="overflow:auto; height:400px; width:400px; border:1px solid black"><h1>Message log:</h1></div>
 </body>
</html>

And next, the server code (written in Python as the title implies) ...

#!/usr/bin/env python

import socket
import threading
import struct
import hashlib
import binascii
import time

PORT = 9876

# evil globals
requestList = []
exitFlag = 0
tic = 0
lock = threading.Lock()
clients = []

# timer tick
class ThreadClass(threading.Thread):
def run(self):
  global exitFlag
  global tic
  global requestList
  global lock
  global clients
  invalidConnections = []
  while exitFlag == 0:
    data = "tick:%s" % (tic)
    data = chr(0) + data + chr(255)
    print data
    # Broadcast tic data to all clients
    lock.acquire()
    [conn.send(data) for conn in clients]
    lock.release()
    tic = tic + 1
    time.sleep(1)

  print "\nNotice - exiting thread !"


def create_handshake_resp(handshake):
  final_line = ""
  lines = handshake.splitlines()
  for line in lines:
      parts = line.partition(": ")
      if parts[0] == "Sec-WebSocket-Key1":
          key1 = parts[2]
      elif parts[0] == "Sec-WebSocket-Key2":
          key2 = parts[2]
      elif parts[0] == "Host":
          host = parts[2]
      elif parts[0] == "Origin":
          origin = parts[2]
      final_line = line

  spaces1 = key1.count(" ")
  spaces2 = key2.count(" ")
  num1 = int("".join([c for c in key1 if c.isdigit()])) / spaces1
  num2 = int("".join([c for c in key2 if c.isdigit()])) / spaces2

  token = hashlib.md5(struct.pack('>II8s', num1, num2, final_line)).digest()

  return (
      "HTTP/1.1 101 WebSocket Protocol Handshake\r\n"
      "Upgrade: WebSocket\r\n"
      "Connection: Upgrade\r\n"
      "Sec-WebSocket-Origin: %s\r\n"
      "Sec-WebSocket-Location: ws://%s/\r\n"
      "\r\n"
      "%s") % (
      origin, host, token)


def handle(s, addr):
  global exitFlag
  global lock
  global clients
  # acknowledge the connection
  data = s.recv(1024)
  s.send(create_handshake_resp(data))

  while exitFlag == 0:
      print "Waiting for data from", s, addr
      data = s.recv(1024)
      print "Done"
      if not data:
          print "No data"
          break

      print "%s Bytes of Data recvd from %s --->%s" % (len(data), addr, data)
      data = data[1:-1]
      data = chr(0) + "msg:" + data + chr(255)

      # Broadcast received data to all clients
      lock.acquire()
      [conn.send(data) for conn in clients]
      lock.release()

  print 'Client closed:', addr
  lock.acquire()
  clients.remove(s)
  lock.release()
  s.close()

def start_server():
  s = socket.socket()
  s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
  s.bind(('', PORT))
  s.listen(1)
  while 1:
      conn, addr = s.accept()
      print 'Connected by', addr
      clients.append(conn)
      threading.Thread(target = handle, args = (conn, addr)).start()

t = ThreadClass()
try:
t.start()
start_server()
except (KeyboardInterrupt, SystemExit):
print "\nNotice - CTL+C Received, Shutting Down, 1 Sec ..."
exitFlag = 1
time.sleep(2)
# note - should probably use join here
print "\nAll Clean, Exiting Server."

That's it. Its just that simple.