While working in a technology company, a little careless mistake can ruin the entire working system.
A user asked the forum, “What’s been your biggest stomach drop mistake in your technology career?” Here are the top responses.
MADE A WRONG SETTING CHANGE IN A DRIVE
“Made a setting change in OneDrive that prevented everyone in the organization from using it (and we use it all day long). I reverted the setting, but it took 3 hours to propagate.
It’s embarrassing but a good lesson in change management.”
MADE A WRONG CONFIGURATION CHANGE
“I made a configuration change to one of my client’s SIEM (Security Information And Event Management) and didn’t view the logs as I’ve done it 1000 times, and it was a minor change (literally one line); I went on vacation, and it stopped log ingestion to my client’s SIEM for an entire week.”
TURNED OFF THE WAN PORT OF A ROUTER
“I disabled the WAN (Wide Area Network) port of a router I was working on 1000+ miles away with no redundant port or LKG (Last Known Good) in place.
Shut off the internet to several people (300+) by accident. I was very new and had to get someone on-site to help me fix it.”
DELETED THE LIVE COMPANY DATABASE
“I was working on a database at Novell (yes, I am that old) that ran on DEC Vax (Digital Equipment Corporation Virtual Address Extension). I deleted the test database, but I deleted the live company database.
The phone calls started hitting my desk within 3 seconds and did not stop all day. It happened at the end of the quarter when everybody was trying to get orders entered. Somehow, I did not get fired.”
NUKED THE LAST DOMAIN CONTROLLER IN A PRODUCTION DOMAIN
“I nuked our last domain controller in a production domain by accident. I rebuilt the domain from backups in about 5 hours. It’s a good thing it happened at 10 PM, and I had it back by 6 AM. Moral of the story. Slow down.”
SHUTDOWN A REMOTE SYSTEM
“Not me, but an admin I worked with many years ago. We worked system support for the United States military from the DC (District Of Columbia) area but were tier 3 support for systems worldwide. Our servers were all running Solaris, and we often had multiple terminal windows open to different servers.
Long story short, the admin meant to shut down a system in our laboratory, so they did an initialization 5 (shutdown, then power off to the OK prompt). Unfortunately, they were in the wrong window and did this to a remote system in South Korea on a mountaintop overlooking the DMZ (Demilitarized Zone). Three hours from the nearest local support technician (also 3 AM local time).”
ERASING AND RELOADING WRONG SWITCH STACK
“Erasing and reloading the wrong switch stack and taking out the company’s biggest office with all the C-level staff.
400 staff at a remote site with no on-site IT (Information And Technology) that day. This was done at 9:30 AM on the company’s biggest sales week.”
REPLACED THE WRONG DRIVE
“It wasn’t me directly making a mistake, but I was remote support for the on-site tech. This guy went into a massive financial institution’s data center to replace one failed drive on an array.
He replaced the drive and left. He didn’t need to call me to let me know he arrived on site, so I hadn’t done anything.
Phone calls start pouring in. Everything’s down. I called the technology department, saying, ‘Yeah, I think I pulled the wrong drive, but I stuck it back in and replaced the right one.’
Nope. I did pull the wrong drive, but then another wrong drive – replaced this good drive with an empty drive and went home with the bad Pen drive still in the array. So the array had the one hot spare trying to take over for the 1st pulled drive. The 1st pulled drive was reinserted; the geeking dead drive was still there, and another pen drive full of good data was in his van.
That entire storage array had a seizure and didn’t make it. Luckily, they had backups. On that array. No offsite backups. I don’t know the end of this story because they left us.
The guy should have just called me when he got on site. I would have blinked at the dead drive, and it would have been an easy day. I know the array vendor tried to correct it for a solid week and told me it wouldn’t happen.”
STEPPED ON THE POWER CORD OF A MINI-COMPUTER AND MORE
“I’ve had a few pretty big ones. On my first IT (Information And Technology) job, I stepped on the power cord of a mini-computer, pulling it out and shutting down a medical clinic for half a day. This was 1991.
Doing a pen test, I was writing an exploit on the fly for Irix, which accidentally crashed one of their Internet-facing web servers. This was 1998.
I crashed an ASA (Adaptive Security Appliance) firewall and an NMS (Network Management Station) server while doing an on-site pen test. The ASA was between two networks I was given to scan, and for some reason, the admins had UDP (User Datagram Protocol) connection timeouts set at 24 hours.
Yep, the state table is complete, and the firewall is crashy. I knew which server I had crashed as it was the only one of its kind, so I tailgated my way into the data center and rebooted/fixed it around 1998.
In 2005-6, I moved some DMZ (Demilitarized Zone) vlans on some switches and production firewalls and briefly took offline an entire alarm receiving equipment. (Worked at a major security/alarm monitoring company) I quickly realized what I’d done, panicked, quickly fixed it, and immediately went to the bathroom and took a massive dump.”
CHANGED PERMISSIONS ON THE GNU PRIVACY GUARD KEYCHAINS
“I accidentally changed permissions on the GPG (GNU Privacy Guard) keychain private key files to make it unreadable on a core server service with thousands of end users.”
USED THE TEST PROFILE TO TEST MOBILE DEVICE MANAGEMENT SOLUTIONS
“I was testing an MDM (Mobile Device Management) solution for my new job, and silly me used the test profile set up to test encrypt a test Phone. Turns out they had added the managing director’s phone to the test profile before I joined the company, so his phone was now encrypted, and to make it worse, he was using it as a personal phone, too.
His photos were erased upon encryption. They got a telling-off for that one. Who is stupid enough to name a profile test when there’s a live phone on it? ”
MIS-TYPED BLOCK LIST
“Early in my career, I was working on blocking a phishing site with a very similar spelling to a legit Microsoft domain. I mistyped it in the block list and unintentionally broke access to almost every Microsoft product in the company.
Luckily, it was caught and fixed fast after complaints started flooding in, but it was the biggest ‘oh god, I messed up’ moment I’ve had so far.”
DISABLED TRANSPORT LAYER SECURITY
“It was probably the time I, in a good faith effort to keep my company secure, disabled TLS (Transport Layer Security) 1.0 and broke a bunch of production stuff that had to be fixed over the weekend.”
FORGOT TO ENABLE BITLOCKER
“Forgetting to enable BitLocker or add a laptop to the correct OU (Organizational Unit) in AD (Active Directory). It’s still early for me.”
CHANGED THE MAIL SERVER
“I was working on our SPF (Sender Policy Framework) records and changed the mail server from Google to Office 365.”
Brother Furious As Sister Declines His Invitation For “Overflow Wedding”. Says, “He Needs To Make The Bride Happy”. Netizens Think He Is Crazy. You Decide
Brother’s wedding is an important event but can there a reason big enough for you to miss it? You decide who is a jerk.
Sister Refused To Host Nephew’s Birthday Party At Her House. Says “He Isn’t Allowed In My House”. We Think She Has A Point.
Is it reasonable for a woman to refuse to host her nephew’s birthday party at her home because she was unhappy with his behavior there? You Decide.
Woman Refuses To Move Aside On The Hiking Trail For A Group. Says, “You’re Rude”. Husband Requests Not To Escalate The Situation. Is He Right?
Is it okay to walk casually in a large group while you’re on a hiking trail and not care to make space for other people? You decide.
Groom Shocked As To-Be-Bride Forgets About His Daughter’s Peanut Allergy. Says, “I Don’t Listen Well”. Thinks She is Right.
How would you respond if your negligence resulted in a child’s hospitalization and a wedding cancellation? An internet user asked, “Am I a jerk”. You decide.
Father Demands Daughter Split The Inheritance Equally Despite His Mother Giving Everything To One Granddaughter. Says, “She Intentionally Set It Up For The Daughter To Fail In It”. Is This Reasonable?
No matter how logical you think you’re being, questioning a dead person’s will and wanting to make changes as you please is not at all… read more.
This article was originally published on Mrs. Daaku Studio.