|
INTEROP LOCATIONS: LAS VEGAS • TOKYO • MUMBAI • NEW YORK |
So Hawaii just went through a tense couple of hours as we waited on the data to come in off the mid pacific buoys in regards to a possible tsunami from the 8.3 earthquake in Samoa. What this really started me thinking about was business continuity and just how would my school, my lab or my home recover from a natural disaster? Well no tsunami, but lots of folks were thinking about “what if”.
So as I’ve been diving ever deeper into the virtual world I’m realizing it’s the storage cost that’s the barrier to adoption rather than the servers. The reality is that production quality storage systems are expensive for a reason. (management, reliability, migration, interoperability, etc) However, how does one get across the training bridge from simple VM Hosts to a Virtual environment with a Storage Area Network (SAN)? The answer is open source, and the flavor I’m playing with right now is OpenFiler and I’ve installed it on a Dell PowerEdge 2800 server that I had lying around the lab.
My overall goal has been to retire all my older inefficient servers and slowly migrate over to higher efficiency blade servers. The downside has been the huge bite in my budget that a fiber channel or iSCSI SAN is going to put me back. Why a SAN and not just use my internal disk drives? Easy, VM mobility and how a VM can move from a shared blade to a dedicated blade to multiple blades that are load balanced. This kind of juggling act only comes from the VM’s being stored on a SAN (of some sort) that can be readily accessed by all the computers in the cluster.
The InteropNET is billed as the world’s largest temporary network and over the years has visited places like: Tokyo, Sydney, Frankfurt, Paris, Mumbai, Atlanta, Las Vegas, San Francisco, and many others. At one time the InteropNET had a seven city world tour. It’s made possible by a cadre of volunteers (YES, VOLUNTEERS!) that design, build, run and then tear down a network similar in size to a large office building. These folks have all applied with a mini resume posted to the volunteer site, then the InteropNET staff will pick the folks that fit the needs of the team. We’re looking for skills, but also willingness to work with the rest of the team, and ability to play well with others. The message you really need to take home on this is that you’ll be working with some of the best in the industry, people that have written a very large number of the Internet RFCs and have designed a massive number of new Internet technologies.
Even with all the gear that the InteropNET deploys, why do we even need a colocation facility, much less two of them? Well the answer is that “you can never tell”. You can never tell if someone is going to do a “backhoe fade” on us, you can never tell if CNN does a landmark article on us and our externals become inadequate, you just can’t tell. While I’m sure the marketing folks would correct me, but in all reality modern colocation facilities might as well write down their bandwidth capability as an infinity symbol. Yeah, I’m exaggerating, but there sure is a whole heck of a lot of bandwidth available in a modern colo. Even the InteropNET can’t afford that kind of bandwidth, so it just makes sense to put some of our services offsite and then utilize geographic load balancing to send inquiries to either Denver or San Jose Cyber Centers.
Let’s look at a scenario for say Fergenschmeir Corporation’s public website. Their website provides product information, documentation, and video tutorials to their customers worldwide. So instead of trying to build a data center in their downtown headquarters, they instead work with Qwest to put their public servers in the Qwest Cyber Centers in San Jose and Denver. They also put some Coyote Point geographic load balancers in both Cyber Centers so that customers are sent to the colo nearest to them. Since Fergenschmeir is running VMWare, they really have no need to physically access the colo (well except for equipment upgrades, etc) and they didn’t go through the expense of building and maintaining data centers in multiple cities.
Apologies to Qwest, I just can’t do justice to what a Cyber Center really can do, so why don’t you watch a bit of video on the topic and let them correct my ranting.
My greatest wish….lock on target…anonymous photo that’s been floating around on the net for years.
VMWare is joining the InteropNET team again this year and helping the NOC to be a bit greener, while increasing the overall reliability of the network services we’re providing. The overall concept is that we create virtual machines just like you would sysgen a normal machine, but instead of installing directly to the metal (i.e. installing onto a new physical server) this time you’re installing under the VMWare management system. These images can be started, stopped, paused or moved depending upon the business rules you’ve defined. We just happen to be running our virtual machines (VM’s) on a blade server, but you could just as easily run them on a mixed bag of legacy servers that you already have. In fact, I’ve been consulting with a couple credit unions on using VMware to leverage their existing infrastructure while still expanding the types of services they provide.
Heck, add in a Coyote Point load balancer and you can load balance, auto-stop/start VM’s and all kinds of stuff. Sure is sounding like a greener future.
This diagram shows how the VMWare virtual infrastructure provides a way to share resources, but the real story is just how green this really is. Ok here’s the story: take a good hard look at your web servers. I betcha they’re probably only running in the 7-20% utilization range. Now look around the back and they might be using anywhere from 700watts all the way up to monsters sucking down over a 1500watts. Now consider this, even the new energy star power supplies don’t reach maximum efficiency loafing along…they actually get more efficient as the load goes up.
Now what if we take those very same servers, toss on a virtualization layer, and now run say a half dozen web servers on it? Each project group (assuming each project group owns their own web server) still gets roots access to “their server” and can still patch it and add to it as they see fit. However, now with a half dozen servers running, the underlying server is now probably running into the 60-70% range….aaaahhhh…much more efficient and we’ve just turned off perhaps as many as five 700watt power supplies. VERY GREEN.
Green enough that Pacific Gas and Electric has an incentive program going to help folks jump into virtualizing their environment, AND save a whole heck of a lot of bucks.
While I didn’t manage to snag some video of the VMWare engineers while at hotstage, VMWare does have some mighty fine podscasts talking about the technology.
Back in the day, a network engineer commented that “…we sure had to run a lot of wire for wireless…” The gist being that we needed to put in a whole heck of a lot of wireless access points to support the loads found at a technology trade show. Now this is NOT to say that the InteropNET venue is by any means “normal”, because it isn’t. From the NOC we can typically see hundreds of access points with people stomping all over each other with their AP’s cranked up to full volume in a feeble attempt to drown out their neighbor. We also have one heck of a lot of territory to cover since our guests will find a quiet spot just about everywhere in the convention center. So our challenges for wireless are: (1) the need for LOTS of coverage, and (2) the need for intelligent coverage so that we don’t add to the “shouting match” between the hundreds of access points in the exhibit hall.
Our partners at Xirrus are back with their flying saucer WiFi arrays, hovering over the exhibit hall, and on tripods throughout the Mandalay Bay Convention Center. However, these flying saucers actually have up to sixteen radios in each saucer, with sectional antennas and patented signal guides to more intelligently direct the signal to where it’s needed instead of just blasting in a 360 degree circle. The end result for the InteropNET is a whole lot less wire run to support wireless and instead of a hundred or so access points, we’re talking about a dozen now.
These “sector” antennas are mounted along the rim of the saucer, each one covering a piece of the “pie” so to speak. With each radio being individually controllable, the Xirrus folks can “shape” the radio energy to better cover odd shaped areas. One of their favorite examples are schools and how they tend to have a high density of users in long narrow buildings.
These anonymous “heat maps” are actual measurements from various Xirrus customer schools….the funny thing is that warehouses also have heat maps that look like these but for a different reason. Your typical warehouse has long rows of metal shelves with stuff on them and the signal from traditional access points will bounce around like billiard balls causing all kinds of multipath interference. The Xirrus solution of shaping the RF energy patterns, greatly reduces multipathing allowing more throughput with less energy.
To get a better glimpse of the technology and Xirrus’ role in the InteropNET, I shot a little video of John Merrill at hotstage where he shows off the guts of an array.
To say that Fluke Networks has been a long time supporter of the InteropNET is a BIG understatement, and while the video I shot of Neal Allen at HotStage mentions a few things, I’d like to tell a few more stories about engineering prototypes that have shown up to the show with developers.
First off, Neal Allen of Fluke Networks has literally written the book on Network Maintenance and Troubleshooting and has been part of the InteropNET team since 1993.
or wait just a bit for his second edition where he’s added in a whole bunch of real world experience from Fluke Network field engineers, customers and the InteropNET team.
So speaking of real world experience, Neal has managed to coerce quite a few of the FlukeNetworks development team, field engineers, TAC engineers, and even some sales folks to help out building the InteropNET. I remember various pieces of cardboard wrapped instrumentation that would magically appear (we’re all under a blanket NDA) so that the developers could try it out. Being the opinionated group we are, we’d poke holes in their “baby”, they’d go back to their hotel room only to return the next day with a new version for us to try out. What’s super cool about the InteropNET is the sheer amount of new gear showing up on the network. Fluke has consistently shown up to try things out, get their feet wet, and get the gang to voice their opinions.
This is a company that firmly believes in “eating their own dog food” as we helped them build a temporary TAC in the Las Vegas Hilton one year when the show was across the driveway at the Las Vegas Convention Center. This was to give the TAC personnel a chance to use the Fluke Networks gear in real life, but also to demonstrate that Voice over IP was a real technology and ready for prime time.
The Fluke Network tools are an integral part of delivering the InteropNET to classrooms, offices and the exhibit floor on time and operational. Each network drop on the exhibit floor is tested a minimum of three times. First for cable integrity, next for network connectivity within the InteropNET and lastly confirmation that you can get out to the world. This attention to detail means that we can put a high level of trust in the InteropNET and that makes it significantly easier to troubleshoot when hundreds of other networks in the vendor booths plug into out network during setup. Having that level of trust is how we can build the world’s largest temporary network.
Coyote Point is back to provide load balancing for the various servers in the InteropNET. Our friend Sergey Katsev tells the story about Coyote Point’s role in a video that I shot at HotStage, but that only scratches the surface of what this talented box can do for your corporation. Concentrating on the nuts and bolts of modern load balancing the Equalizer has the ability to start at legacy layer 4 service swapping (i.e. if it’s TCP post 80 web traffic, just switch it back and forth between available servers) but the story goes quite a bit further into layer 7 load balancing where services like SSL acceleration, global/geographic load balancing, virtualization control, and service health detection all become possible. Load balancing is key to giving your IT staff maintenance windows (shift the load while servicing a machine), leveraging your IT dollar (use weighting to determine which server can handle more load as you get more use out of legacy servers), and save on WAN links by shifting your load to a colo nearer to your customers.
Towards the end of the demonstration, Sergey is showing how the Equalizer works in the VMWare virtualized environment by purposely shutting down one of our SMTP relays to simulate a server failure. The result is the equalizer connecting to our VMWare cluster and automatically starting up our backup SMTP relay hidden away in a separate set of racks outside the NOC. So let’s take a short flight of fancy and imagine a famous brand lingerie show, whose first showing managed to overload and crash their ISP. What they could have done is setup an equalizer in front of servers at key spots around the globe and let the global load balancing features direct potential customers to the closest spot. They could go further and have a VMWare virtualized environment at each colocation facility which could be setup to increase the number of virtual servers according to actual customer load. The Coyotes have a great paper explaining Geographic Load Balancing in detail. They’ve made a pretty big collection of education and marketing materials available in their document center along with some flash video on how it all works.
The InteropNET just happens to be using six of these boxes: two in the NOC, two in a backup NOC, and two more in the Qwest Colorado Colocation facility. Attendees asking for information from outside of Las Vegas may actually be getting the info off servers in Colorado instead of the NOC, and it’s all controlled by those clever coyotes. Come see them in the NOC, just look for the bright red paw print in the rack.
NEMA, L6-20, L5-30 and other arcane words like those will usually pass through the average network engineer without leaving a trace. But in the InteropNet people are learning fast that some of the basic requirements can’t be ignored :
Devices need power, and the plugs had better fit.
Of course, this isn’t always the case, and this year we have had an unusually high number of ’square peg, round hole’ issues . Time isn’t on our side with HotStage, so waiting another 5 days for someone to ship a replacement isn’t going to let us configure out equipment. So we often have to turn to less conventional methods to get out work done. If high voltage power scares you, you should probably look away now.
Interop Tweets
Sep 29th, 2009 |