![[SimGuy.net]](/shared/logo.png){kind=logo}
June 2005 Archive
Thursday, June 30, 2005
[Comments: 1] 6:36 AM - balancing-act
Following the timeline from last Friday night through right now, I have been unbelievably busy, and I'm not getting anything on my personal todo list done.... well, I managed to get a haircut and mail my dad's birthday card, but other than that, it's been pratically all work.
Friday was the power outage and related chaos. Each subsequent day, I ended up having something to do at the datacenter, and I ended up spending far more time there than I wanted to. I just finally hit level 40 in World of Warcraft, and I've had less than 4 hours to play this week.
At the beginning of this week, I got my first chance to learn about setting up a load balanced server. While interesting, this learning curve had no choice but to be about 3 days from start to production.
I actually spent an overnight working on trying to learn how the system worked and getting the three servers prepared. I went down to the datacenter at about 12 AM on Monday morning, and I stayed there until about half past noon. I got the system basically working, and I was very proud of myself, but I didn't end up going to bed.
I decided I needed a haircut badly that morning, so I went looking for a place to get one after I left the datacenter. I found some listings online in 2100 S Indiana, which is a maze of an office building. I got fairly lost in the building, and though I did find all three places (I also found three separate elevators), they were all closed. I ran into a guy while I was leaving who asked, "are you trying to find your way out of here?" which I think demonstrates the problem with the building's floor plan. I asked him about the hair cutting places, and he said they're all closed on Monday. You'd think at least one of the three would be open!
Anyway, I did eventually find a place to get my hair cut that day, though it was very expensive, and the guy didn't seem to like using the trimmer, so he cut my hair with scissors quite impressively, about as short as the trimmer would have done. For the "show," I think I can accept the higher cost. :)
Since I never got to bed on Monday morning, and Karl managed to somehow keep me up until 5 AM on Tuesday morning trying to set up MySQL on the load balanced system, I ended up running about a 36 hour total run without sleep. I spent a good deal of Tuesday snoozing. Somehow I ended up at the datacenter again Tuesday night, twice, for a total of about 4 hours doing random work. At least I was running on some sleep by that point....
As for Wednesday, production day for the load balanced system! We had to do all final data transfer, testing, tweaking and setup to get the sites live by late evening to ensure a smooth move from the customer's previous host. It went pretty smoothly though we are still monitoring the servers for a number of possible problems to ensure the load balancing is working and to make sure that all the software is properly optimized. That took care of my afternoon and evening.
We got out to Lou Malnati's for pizza. I haven't had Lou's in a while, and comparing it to Pizzeria Uno/Due, Lou's is enormously better. Next time my parents get out here, I have to make a point to take them to Lou's for a better overall experience. :)
Anyway, after dinner, the load balanced servers started to give us some trouble, which basically translates to our constant monitoring and tweaking to try to fix their glitches. But worse, VPS1 went haywire. We requested some emergency support from SW-Soft to try to figure out what caused all the virtual servers to just essentially lock up while the operating system continued to function as if nothing was wrong. SW-Soft developers are still analyzing data, and we've established some extra debug logging in case it happens again.
I am hoping my Thursday and Friday are more tranquil. I'm totally ready to relax. I want to play WoW, and I need to at least have some time to call my father for his birthday. :)
Happy birthday, Dad! :)
Saturday, June 25, 2005
[Comments: 0] 9:26 AM - can't-stop-thinking
I just had a thought. I bet the rumbling sound was the sound of every large metal (garage door-style) fire door closing in the building at once.
[Comments: 0] 9:24 AM - afterthought
I forgot to mention.... just after the power went out here at our apartment there was a horrible rumbling sound. It sounded vaguely of thunder, but lasted way too long, and there were no storms anywhere nearby. I haven't got a clue what it was, but it was kinda disturbing....
Also, the hall lighting in our part of the building didn't work (not even exit signs, which we will report), so it was navigation in the dark. By the time I came back here, power was back on, but apparently all the heat alarms on the 2nd floor were tripped and all the fire doors are closed.
[Comments: 1] 5:54 AM - great-chicago-blackout-of-2005
The power went out! Much of Chicago lost power last night after some random flickering earlier in the evening. I was playing World of Warcraft with Andrew when the power went out suddenly and completely.
While my battery backups held for a few minutes (proving that they did work), the power had clearly decided it was not coming back on, and since we had also lost Internet access, I ended up just shutting the computers off.
On the topic of power backups, we decided without power, that we might as well wander outside to see what's going on. It's been hot as hell here (greater than 95 degrees for highs the past two days), so we ended up figuring we might as well head to the datacenter, where there's guaranteed to be power, air conditioning and Internet access.
Wrong. Equinix battery backup and generators were not able to keep the facility running. On more than one occasion all of our equipment lost power and power cycled knocking us (and all the other hosting providers, bandwidth providers, telephone companies and other telecommunications companies) out for several minutes while our equipment recovered. Now I'm staying down at the datacenter until 8 AM, at which point Ray will come in and take over for me. We want to keep people here until we're sture everything has stabilized completely and all customer equipment is back up.
This is something that isn't supposed to ever happen. The state of the art power backup systems are supposed to prevent our equipment from ever losing power. Instead we effectively had several small blackouts here. Equinix is going to be in deep trouble as they find on Monday that every one of their customers is going to be looking for some compensation for this failure. 100% power reliability is one of the things we pay a lot of money to be guaranteed.
So, anyway, apparently the power failure occured as a result of a fire at a sub station somewhere in Chicago, but the details are still sketchy. From news reports, about 38,000 people lost power. The news reports don't mention what this outage did to the Internet connectivity in the upper midwest. The scale is massive, but fortunately the recovery has been reasonable.
Back to work for me, for now. Ray said power wasn't back on at the apartment when he and Karl got back. Hopefully by the time I get in later it will be, and I'll actually be able to play World of Warcraft for a much longer time tonight. :)
Thursday, June 16, 2005
[Comments: 1] 4:55 AM - driving-habits
I recently discovered something about my skills as a driver. I am much better when I am driving by myself. When I have other people in the car, I ask them for directions or confirmations, I get confused, I make strange mistakes, and overall I feel disoriented.
When I'm driving by myself (or under the certain conditions where I know that the people in the car don't know anything that might help me), this doesn't happen. I don't have anyone to ask, so I follow the directions as I remember them, I make decisions more quickly without second guessing and I tend to have a much clearer recognition of where I am going, how to get there, and what I need to be doing.
I wonder if I can train myself to pay less attention to my passengers. :)
[Comments: 1] 4:48 AM - catch-up-game
I guess I figured I'd just ramble about some random stuff that happened recently so that whoever might be paying attention might have something to read, so here we go....
So first, we moved! June 1 marked the day we began transporting all of our items from the apartment in Evanston to the apartment in Chicago. It's a nice, huge place, and though it took a good while to move all of our stuff, including many trips in my car, we are now mostly settled.
We gave up on a number of items from the old place, including our previous couches. Actually, those were the main items of significance. As our booth at HostingCon required, we traveled to Schaumburg, got lost, drove around for 30 minutes, and finally found Ikea, past closing time. We returned later to review what we wanted to buy, which was a great deal of furniture, and I decided it was a good idea to make a list. We returned again the next day finally to buy the items we needed and transport them to the convention center and/or apartment. This was last Sunday (the 5th). The convention was Monday and Tuesday, so our set up Sunday night accounted for most of the booth planning and coordination we did, something which we must repair next time with additional advanced planning, though hopefully we won't need to buy quite so many things the next time around.
I met some rather interesting people with interesting products, including the creator of the H-Sphere control panel that we use for our shared hosting system, and the president of a company that makes a rather awesome hosting control panel we'd like to offer to customers called InterWorx. I was far more excited to be wandering around evaluating potential software than talking to potential customers, since I'm definitely no sales rep. Unfortunately, I didn't have any business cards, so no one is going to remember who I am come next convention... :)
We had a bit of a crisis during the convention as a variant of the mytob email worm started running a phishing scam pointing to one of our IPs. Even after we suspended the account on our network, we continued to receive abuse complains in droves, up to several hundred via email, and dozens of phone calls over the course of the next few days... and this was all after we already took action, and no longer had anything else we could do. Apparently the FBI has stepped in and wants to run an investigation, so apparently the scope of the incident was big enough to raise flags. It was bad timing for us, without Internet at the apartment and having our attention diverted by a convention.
This move has been very exciting. I really like this idea of living in Chicago rather than a suburb. I think best of all, it's not right in the middle of downtown, so it's not the "horror" I had once thought it might be living in a major city. It's rather quiet around here, the neighborhood seems pleasant, and I feel relatively comfortable.
At the same time, the move has me feeling a bit overwhelmed and disoriented. I think transitions often do that to me. I often end up reflecting backwards a lot more after I make a transition of this magnitude and I've consequently been thinking a lot about what I miss from college. I'm saddened by the fact that stuff is all done now. I'm still trying to find the right way to build myself more of a local social life, so I can at least find some other places to direct my recreational needs. Playing World of Warcraft and watching TV don't always cut it. :)
Tuesday, June 14, 2005
[Comments: 0] 4:33 AM - soggy-fun
If you've been paying any attention in the proper context, you've probably noticed that I got back online the day after I posted that last blog entry. We played catch-up after the convention and restoration of our Internet connectivity, and things are going pretty well.
Now Ray's on vacation, so Karl is filling in for many of the things he usually does. I'm trying to start spending more time down at the datacenter to offset some of the time Ray would previously get roped into spending there because I didn't want to make a 45 minute trip downtown at 3 AM. I think that is going to translate to many more nights where I spend an hour or two working down there, which doesn't bother me really, except that working among the noise of a billion (note: exaggeration... maybe) fans blowing can be a bit numbing after a while.
I got to hang out with my cousin yesterday. She visited the new apartment, seemed to like it, and then we headed down to be tourists and wander around Navy Pier (and eat lunch). Overall overpriced and somewhat interesting, we both ended up finding a nearby park full of odd artwork worthy of more conversation. Still, it was nice to finally get to be a tourist for a bit, and it was great spending some time with my cousin.
It looked like rain a good deal of the time we were at Navy Pier, and even drizzled a few times, but the storminess finally hit right when I was coming back from the datacenter at around 3 AM. It held just till I reached the building, and then it picked up a bit. While I wanted to keep my laptop dry, I felt like enjoying the rain, so I brought my bag inside, and then wandered back out to get wet for a little while. A couple nice flashes of lighting complemented the scene, and I stood outside for at least a good 20 minutes. I think I'm mostly dry now. :)
Thursday, June 9, 2005
[Comments: 0] 4:13 AM - missing-in-action
As you have probably noticed by now if you spend any amount of time each week communicating with me, I haven't been online in a good while. This is because after returning from Long Island, we prompty moved to Chicago on June 1st.
At some time early that day, SBC disconnected our telephone in the old apartment, effectively terminating our DSL. While annoying, it was not entirely unexpected, as we were moving anyway, however we then expected to have the phone line set up in the new apartment within the next two days. This never happened, and thus we've managed to retain total lack of access to the Internet. When we called SBC, they told us the line was set up, and it does ring when you call it, so we're suspecting the building may be at fault.
However, with the hectic weekend and HostingCon at the beginning of the week, we've hardly had the chance to get anything else worked on.
We're taking care of support requests a couple times a day from the datacenter, and that's where I'm writing this blog entry. However, our response times and our customers are suffering a bit because of it.
Hopefully we'll be able to get everything worked out soon, and I'll finally be back online. As mentioned in my previous post, my cellphone is a great way to reach me if you have a burning desire to talk. I'll also answer if you email me... more than likely. :)
