bill_sffcuMemberAug 07, 2009 at 8:58 am #143648
Over the last few months we’ve been moving our branch locations onto a fiber optic based layer 2 network as our T1 contracts end. Along with the new connections we’ve been installing new 2811 routers to replace our aging 2621s.
Because this was a phased migration we had to keep some older equipment in production until all the T1s were taken down. Our core router, a 3640, located at our main branch had all the T1’s terminating into it. Once we started hooking up the fiber we installed a 2811 at our main branch and ran it in parallel with our 3640.
Just last week we got the last branch moved onto the fiber. Depsite that move the 3640 was still doign some work routing traffic to a few third party routers we have at our site. It was also acting as the default gateway for all of our clients.
With the new 2811 at our main site having all the static third party routes added to it my plan was to wait until our branches closed and then change the 3640s IP to something different and then have the 2811 take the old IP of the 3640. That way I would not have to worry about clients and other third party devices (routers) having to change their default route.
The plan seemed to work out well. I tested several sites including our third party apps and they all worked. With that done I went to punch out. Our timeclock is a web based app that we access via the internet. For some reason it was not loading properly, but the internet was working just fine. I had done some browsing to make sure of that when I did my testing. I chalked it up to the webclock being down yet again and sent my boss an email to alter my time.
That all happened last night. This morning, about 30 minutes before I usually get up my boss called me saying that some of the early morning staff were having problems accessing sites. It was all very intermittent and inconsistent. One person could access this secure site whereas the next person could not. Internet access seemed a bit sluggish as well. I tried to VPN in but for some reason I could not access anything on our internal network so I got ready early and headed straight into work. The fact that I made a network change the previous night pointed me in the direction that I had messed something up. But, we could access the internet. It was just flaking out on us randomly and secure sites in particular were having difficulties. We use Websense for internet filtering so out first step was to shut that server down to eliminate it as a possible cause. At the same time I logged into the 2811 at our main site and began looking through the config fully expecting not to find anything.
This is what I find.
ip route 0.0.0.0 0.0.0.0 172.16.1.1
ip route 0.0.0.0 0.0.0.0 172.16.1.2
1.1 is the internal address of our ASA
1.2 is the old address of our 3640 which is now the address of the 2811.
With the 2811 running in parallel with the 3640 I gave the 2811 a default route of the 3640. When I decommisioned the 3640 I added the ASA as my default route while failing to remove the old default route.
Usually when I’ve messed something up on a router one or more networks become completely inaccessible which immediately tells me that I’ve done something wrong. With this mistake the problems were very intermittent and seemed to point away from the router as being the problem. Once I removed the extra default route the problems immediately went away.
You must be logged in to reply to this topic.