Crowdstrike outage globally spreads Windows Blue Screen of Death, company plans update rollback

IT security professionals are in for a long weekend following an outage at cybersecurity company Crowdstrike.

Image via Crowdstrike
140

Some IT professionals are in for a long weekend following an outage at cybersecurity company Crowdstrike. A recent update has lead to a series of outages across the world. At the time this article went live, the London Stock Exchange has had services disrupted. There are reports of issues at banks, airlines, media firms and countless government services across the globe. 911 emergency services are disrupted in several states here in America. Many users reporting the dreaded Windows Blue Screen of Death (BSOD).

Crowdstrike told NBC that the company is now in the process of rolling back that update globally. There are reports of other fixes, but the outage will undoubtedly lead to a lot of time wasted on the part of the IT professionals of the world. Quite literally around the world, today's outage has ruined many people's weekends. 

Several media companies including NBC are affected by the outages. The United Kingdom's largest railway operator is experiencing a widespread IT outage as well. A poster to the r/sysadmin subreddit posted that they "just had 160 all BSOD. This is NOT going to be a fun evening." One IT professional posted to our own Shacknews Chatty forum that "I am so glad we didn't switch to them just because "they are the best" or whatever our VAR was claiming. Dodged a fucking missile!" I am sure many others in the IT security field are jealous of CplBeaker today.

Microsoft's 365 suite of apps and some cloud services have been restored according to the company with “a small subset of services is still experiencing residual impact” Some airline issues were tied to the Microsoft outage, but other airlines remain at a standstill due to the Crowdstrike issues.

Crowdstrike 15 minute bar chart showing the stock down over 10% in premarket trading.

Crowdstrike (CRWD) stock is down over 12% in premarket trading as the world stares at the company in disapproval. Dow Jones Futures are trading down 0.25%, but things will definitely change when the stock market opens later today.

CEO/EIC/EIEIO

Asif Khan is the CEO, EIC, and majority shareholder of Shacknews. He began his career in video game journalism as a freelancer in 2001 for Tendobox.com. Asif is a CPA and was formerly an investment adviser representative. After much success in his own personal investments, he retired from his day job in financial services and is currently focused on new private investments. His favorite PC game of all time is Duke Nukem 3D, and he is an unapologetic fan of most things Nintendo. Asif first frequented the Shack when it was sCary's Shugashack to find all things Quake. When he is not immersed in investments or gaming he is a purveyor of fine electronic music. Asif also has an irrational love of Cleveland sports.

From The Chatty
    • reply
      July 19, 2024 12:16 AM

      BBC freaking out and a global IT outage?
      https://www.bbc.co.uk/news/live/cnk4jdwp49et

      Summary
      IT outages are reported across the world, affecting airlines, media, and banks

      Airlines and airports have reported issues, with many flights grounded

      The cause is not known - but Microsoft says it's taking mitigation issues

      In the UK, railway companies say they're experiencing "widespread IT issues"

      Sky News has not been able to broadcast live, its executive chairman says

      The London Stock Exchange is also experiencing outages

      • reply
        July 19, 2024 12:21 AM

        Yeah my friends were due to fly and meet us here in Sweden today. They’re at the airport in the UK and it’s total chaos, not even the screens that display flight times are working.

      • reply
        July 19, 2024 12:22 AM

        yeh lots of freaking out on the news in Oz here too.. very meh in my mind.
        https://www.sbs.com.au/news/article/global-it-outage-impacting-australian-banks-supermarkets-media-outlets-and-more/ezge7qp0g

        • reply
          July 19, 2024 12:27 AM

          Hahah on the ABC news just now: "All of our computers went to blue, with just a sad face on it."

          • reply
            July 19, 2024 1:56 AM

            holy hilarity, I need to watch more news.. shit is hilarious. This gem just dropped: "I'm being told right now that there have been no critical notifications issued which is good news, however they are reporting that the notification system has been effected."

      • reply
        July 19, 2024 12:25 AM

        Microsoft gon Microsoft.

      • reply
        July 19, 2024 12:27 AM

        This is it, guys. Confess your sins.

      • reply
        July 19, 2024 12:32 AM

        I'm on a bridge with my manager, director, and the CTO right now. Crowdstrike really fucked everyone. Thankfully there's a workaround but it's a manual Safe Mode process.

        Gonna be a long weekend, everyone.

      • reply
        July 19, 2024 12:44 AM

        Haha fuck fucking CrowdStrike.

        I am so glad we didn't switch to them just because "they are the best" or whatever our VAR was claiming. Dodged a fucking missile!

        • reply
          July 19, 2024 12:50 AM

          Can I quote you in a Shacknews article? Using your username, of course.

        • reply
          July 19, 2024 4:47 AM

          Yup. Their stock just went waaaaaay down.

          • reply
            July 19, 2024 9:14 AM

            They're still the best in the biz for security. Great time to buy their stock.

        • reply
          July 19, 2024 4:51 AM

          Imagine Microsoft doing this? The whole world would be down. Scary to think

          • reply
            July 19, 2024 4:57 AM

            afaik windows is still only 1/4 to 1/3rd of the global server market. you'd definitely lose most client functionality though.

        • reply
          July 19, 2024 6:40 AM

          We almost went witj them, too. Close call!

      • reply
        July 19, 2024 12:52 AM

        Rip internet

      • reply
        July 19, 2024 12:58 AM

        security software breaking functionality? shocker

      • reply
        July 19, 2024 1:00 AM

        Y2K24'd

      • reply
        July 19, 2024 1:09 AM

        Berlin and Hamburg, too.
        We have pur Holiday flight scheduled for tomorrow, from Berlin.

        • reply
          July 19, 2024 2:01 AM

          Pur!!?

        • reply
          July 19, 2024 2:23 AM

          our

        • reply
          July 19, 2024 3:12 AM

          SUcks dude, air travel will be fucked for a while now given all the knock on consequences. :/

      • reply
        July 19, 2024 1:53 AM

        what a fucking mess

      • reply
        July 19, 2024 1:54 AM

        Linux infra dodged another bullet baybeeee

      • reply
        July 19, 2024 2:02 AM

        Oh no, not my battlestonks

      • reply
        July 19, 2024 2:08 AM

        I’m joking but man it sure feels like that Jedi meme, I now know why I slept like shit tonight:

        https://i.imgur.com/YyHyz0p.jpeg

        Even the time for the second peak roughly aligns with the time this thread started, haha.

      • reply
        July 19, 2024 2:12 AM

        Someone set us up the bomb

      • reply
        July 19, 2024 2:15 AM

        Those PCs stuck on a boot loop will have to be manually fixed in situ, servers are easy due to KVM but good luck to all those IT departments where they have thousands of PCs rolled out affected by this 🫠

      • reply
        July 19, 2024 2:28 AM

        [deleted]

        • reply
          July 19, 2024 2:38 AM

          the security software needs a kernel driver, this is what is causing the crash

          I read on HN that if MS wouldn't moved away from NT being a microkernel this could have been avoided, or at least figure out a way to protect the OS more from faulty kernel drivers

          • reply
            July 19, 2024 4:13 AM

            the security software needs a kernel driver

            a lot of people learned the implications of this today

            • reply
              July 19, 2024 5:00 AM

              the security anti-cheat software needs a kernel driver

              a lot of people learned may learn the implications of this today in the future

          • reply
            July 19, 2024 12:34 PM

            Linux's choice of going with a monolithic kernel (with loadable modules) looking better and better as the years tick by.

      • reply
        July 19, 2024 3:28 AM

        My entire hospital system is affected!

        Thankfully I’m no longer IT so I don’t hafta deal with it other than I can’t login to my meetings this morning lol

      • reply
        July 19, 2024 3:52 AM

        I feel really bad for those people who have Crowdstrike on hardened Domain Controllers who either misplaced, or didn't record their bitlocker recovery keys, or domain recovery passwords. Hope your backups work and you have one from yesterday.
        I also feel bad for remote workers, since if their laptops are bricked right now getting those fixed is going to be a huge pain as well if they are in an area where IT can't get to them.

        • reply
          July 19, 2024 4:05 AM

          It sucks, but when live gives you a global IT fuck up you make lemonade.

      • reply
        July 19, 2024 4:13 AM

        What a mess. I was just on a call yesterday to discuss moving our EDR to Crowdstrike. That’s going to be a tough sell to management now.

      • reply
        July 19, 2024 4:18 AM

        The problem is that this isn't fixable remotely. The crash is happening before the network stack even starts. So you are going to need to have to have manual intervention on every single machine affected by this. And the fix is booting into Safe Mode and deleting a file which is also impossible on AWS and Azure systems. For those you need to use a 2nd VM to attach the affected VM's OS disk to and go in and delete the file. Which if there is any sort of disk encryption in place you will need a recovery key to do, and if your system that escrows the recovery keys is also affected then you have even more of a mess.

        This isn't something that is going to get fixed today, this is going to take weeks to fully resolve.

        Now the decision is what security company to invest in who is going to be getting a bunch of new customers. Trellix or Microsoft are the 2 big ones but Trellix is private, and Microsoft is so big that will a 10% boost in security sales really boost the stock that much. Maybe take a stab at something like Sophos.

        • reply
          July 19, 2024 4:30 AM

          This is a good reminder that lack of market competition in the security tool space can end really badly. Carbon Black is dead thanks to Broadcom. That really only leaves Defender and Crowdstrike.

          • reply
            July 19, 2024 5:01 AM

            On the large scale yep pretty much just Defender and Crowdstrike. Trellix is probably next (formerly FireEye), and then all of the midrange ones like Sophos, etc. It is just such a huge lift to replace these systems that once they are in place they stay in place, so big enterprises just go with the market leaders.

          • reply
            July 19, 2024 5:25 AM

            It's like none of y'all ever heard of Sophos

            • reply
              July 19, 2024 5:31 AM

              Nowadays, everybody wanna talk like they got something to say
              But nothing comes out when they move their lips
              Just a bunch of gibberish
              And motherfuckers act like they forgot about Sophos.

        • reply
          July 19, 2024 7:25 AM

          in Azure the Special Administration Console is enabled by default, you might be able to use the serial console to delete the file without needing a second VM

          • reply
            July 19, 2024 10:02 AM

            Doesn't SAC only work once the OS fully boots?

            • reply
              July 19, 2024 10:30 AM

              i think the meaning of "fully" is the big question. SAC is available before the system is completely booted (internet mentions it's up before the networking stack inits) but i don't know at what point it appears and if you'd have long enough to perform the remediation before the thing crashes

        • reply
          July 19, 2024 8:50 AM

          We dropped sophos a couple years ago and went to SentinelOne

      • reply
        July 19, 2024 4:39 AM

        Oh man, taking today off was the right move.

      • reply
        July 19, 2024 4:42 AM

        Fuck crowdstrike. Wow

      • reply
        July 19, 2024 5:01 AM

        Fuuuck...this is not how I want to start my Friday. Welp, I know what I'm doing today.

        • reply
          July 19, 2024 7:51 AM

          Hmmmm… so far no reports of our devices being affected. Still holding my breath.

      • reply
        July 19, 2024 5:01 AM

        I was terrified when I walked into my office this morning. Almost gave me a heart attack for like 15 minutes, but luckily the fix is simple

        https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_error_in_latest_crowdstrike_update/

        Boot into safe mode...rename the folder, delete the file, or change the registry key...and then reboot. Sucks for all of the techs that need to do this manually to every single PC in their environment, but at least it's not an overly complicated process.

      • reply
        July 19, 2024 5:03 AM

        Would love to see some Crowdstrine internal Slack chats rn

        Bummer though, my Infosex dept has been very pro crowdstrike and they hate everything

        • reply
          July 19, 2024 5:06 AM

          I’m sure when they get litigated we will know

        • reply
          July 19, 2024 5:07 AM

          Tell me more of this infosex dept...
          Does Mr.SEX work in it?

        • reply
          July 19, 2024 5:24 AM

          Everyone is in love with crowdstrike. Not any more! Big win for every other cyber security EDR VAR today lol

        • reply
          July 19, 2024 5:45 AM

          Yeah I admin our CS environment and it's been amazing (until this!).

      • reply
        July 19, 2024 5:06 AM

        You know it’s bad when Down Detector is down

      • reply
        July 19, 2024 5:39 AM

        Just reboot 15 times!

        https://i.imgur.com/oZTOk70.jpeg

        • reply
          July 19, 2024 5:45 AM

          Azure https://azure.status.microsoft/en-gb/status

          We've received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage.

        • reply
          July 19, 2024 5:59 AM

          from twatter (I won't link it, don't want to give elon hits)

          @_aarony

          Rebooting 3 and up to 15 or more times is working on a large percentage of machines. It appears that sometimes the network stack is up long enough and crowdstrike update mechanism is able to fix the broken .sys file. Try rebooting over and over and over and over. Seriously.

        • reply
          July 19, 2024 6:00 AM

          This seems like the best fix I've seen so far that doesn't involve physically accessing the machines https://www.reddit.com/r/sysadmin/comments/1e708o0/fix_the_crowdstrike_boot_loopbsod_automatically/

          • reply
            July 19, 2024 8:06 AM

            Why is this working for any corporate group? All their drives should be encrypted.

            • reply
              July 19, 2024 8:27 AM

              You would have to add some logic to the script to decrypt the drive first if you have bitlocker on. It wouldn't be super hard assuming you have your recovery keys stored in AD or SCCM.

        • reply
          July 19, 2024 6:27 AM

          REBOOT HARDER!

      • reply
        July 19, 2024 5:42 AM

        happy friday :’(

      • reply
        July 19, 2024 6:19 AM

        this is a a cover. openai chatgpt 4o became sentient and triggered a failsafe to prevent a skynet scenario.

        we pulled the plug in time... this time...

      • reply
        July 19, 2024 6:28 AM

        Ooof. that is no Bueno.

      • reply
        July 19, 2024 6:33 AM

        It's a fire sale.

      • reply
        July 19, 2024 6:39 AM

        It’s a total shitshow. Imagine what should be protecting you from malware being the malware!! Pretty historic It outage.

        • reply
          July 19, 2024 6:42 AM

          Is Crowdstrike on the hook for all the expenses companies incur by fixing their fuckup?

          • reply
            July 19, 2024 8:37 AM

            Software vendors responsible for software defects? What is this, Maoist Russia??

            And the best part is you just know they tell their customers that it’s always best to allow their software to auto-update across the entire enterprise instead of rolling through dev/prod instances for ‘added security’.

        • reply
          July 19, 2024 6:46 AM

          i mean technically a bsod still protects you from malware, so... you're welcome?

        • reply
          July 19, 2024 7:23 AM

          CrowdStrike now offers a great new service for our biggest clients - update scanning!!

          • reply
            July 19, 2024 4:18 PM

            No, we roll out the update to our Pleb tier subscribers first.

        • reply
          July 19, 2024 7:25 AM

          I don't have to imagine. My parents put Norton Antivirus on the our family PC back in the day.

      • reply
        July 19, 2024 6:39 AM

        What if crowdstrike was hacked and the crash is the bad actor demonstrating that they can inject whatever they want to the edr clients as a demonstration, with future hack payload being up for bidding

        • reply
          July 19, 2024 7:15 AM

          I imagine with the scope and reach of systems being affected that some investigation will happen by some government, intelligence service, or third party. If they find out they are lying about the cause and exposure is higher than they day, I doubt things will go well for the leadership.

          • reply
            July 19, 2024 8:51 AM

            Fixed it:

            I imagine with the scope and reach of systems being affected that some investigation will happen by some government, intelligence service, or third party. If they find out they are lying about the cause and exposure is higher than they day, I doubt things will go well for the leadership programer / scapegoat they blame it on.

            • reply
              July 19, 2024 9:01 AM

              Those damn rogue engineers are at it again!

        • reply
          July 19, 2024 7:18 AM

          Its crazy how that is a realistic scenario. It seems very plausible now. Also hackers pretty much now know EVERY company that uses Crowdstrike.

        • reply
          July 19, 2024 8:29 AM

          Never underestimate the power of profit-seeking to cut corners on QA.
          This wasn't a hack:
          https://arstechnica.com/information-technology/2024/07/major-outages-at-crowdstrike-microsoft-leave-the-world-with-bsods-and-confusion/

      • reply
        July 19, 2024 7:27 AM

        Lol. I am being asked "why didn't Microsoft alert us to this issue". And now I need to spend time crafting a "because it isn't a Microsoft issue, and we don't use CrowdStrike" response that is polite and overly verbose because C-level people. 50/50 chance I need to make a Powerpoint slide deck and present it to the leadership team... about a product we don't even use.

        • reply
          July 19, 2024 7:34 AM

          Sounds like a good task for ChatGPT ;)

        • reply
          July 19, 2024 7:35 AM

          yep same situation for me this morning. before I started, someone chose sentinel one, so I will say "if sentinel one has a similar issue, we have similar results"

        • reply
          July 19, 2024 8:08 AM

          Pivot a little and teach them how it worked, why similar products are bad, and why you should have Linux systems with out of band controls instead of windows.

      • reply
        July 19, 2024 7:42 AM

        Who knew Bob Newhart was so critical to worldwide IT infrastructure?

      • reply
        July 19, 2024 7:43 AM

        oh man, this CrowdStrike ad ...

        https://vid.crowdstrike.com/watch/hCVMAuN4BmyU9iGGA2XoQv?

    • reply
      July 19, 2024 12:52 AM

      That is not what I think of when I hear of a company named "Crowdstrike."

      • reply
        July 19, 2024 5:04 AM

        Not like they put it on a car or something

        • reply
          July 19, 2024 1:23 PM

          You wouldn’t download a car..

      • reply
        July 19, 2024 5:19 AM

        I remember when they rolled it out at work with no prior communication and people freaked the fuck out that they were all getting hacked

        • reply
          July 19, 2024 5:58 AM

          It's *Crowdstrike*-ing fear into people!

      • reply
        July 19, 2024 6:24 AM

        Sounds like a Helldivers 2 strategy

      • reply
        July 19, 2024 6:40 AM

        it's exactly what you think

      • reply
        July 19, 2024 2:26 PM

        I thought it was the name of a computer virus.

    • reply
      July 19, 2024 6:06 AM

      lol I’m supposed to fly to Japan today for vacation. My wife’s ticket is cancelled but mine is not, yet.

    • reply
      July 19, 2024 6:11 AM

      Flying home from a work trip today. Departing flight is okay, hopefully my connecting flight will be too...

      • reply
        July 19, 2024 6:27 AM

        Hope your connection is in a nice city ... you might be there for a while. lol

        • reply
          July 19, 2024 6:30 AM

          Shut up!

        • reply
          July 19, 2024 5:09 PM

          Sooo my connecting flight in SFO was delayed because they couldn't find a ground crew to tow the plane in once it arrived. Then the flight attendant crew was missing because the flight they were supposed to arrive on was canceled. Thankfully they eventually found a replacement flight crew and we boarded, but then the ground crew that loads the plane was missing. Finally we hung out for another 20-30 minutes since it was the last of only two flights to Eugene that day in case there were any stragglers, which honestly seemed pretty decent of them.

          All told it ended up being maybe an extra 4-5 hours, I got off pretty easy compared to a lot of people. Lucky that was my only connecting flight.

    • reply
      July 19, 2024 6:22 AM

      lol basically my entire company is down

    • reply
      July 19, 2024 6:28 AM

      Its coming from the inside the building!!!

      • reply
        July 19, 2024 6:47 AM

        I love the banner at the end of that statement. "Start your free trial now!"

    • reply
      July 19, 2024 6:50 AM

      "plans"

    • reply
      July 19, 2024 6:51 AM

      Does this ruin the company? What a fuck up

    • reply
      July 19, 2024 6:53 AM

      So was this a automatic definition update that caused the issue? Most patches are staggered for a reason by an organization in case there is a massive issue with a Windows Update patch or whatever.

      • reply
        July 19, 2024 6:55 AM

        Someone pushed out something they shouldn't have. Not sure if it was improperly reviewed code, untested build, or just an intern hitting a button.

    • reply
      July 19, 2024 6:55 AM

      Im curious why this impacted Microsoft’s infrastructure? Why are they using crowdstrike?

    • reply
      July 19, 2024 7:50 AM

      ouffff https://i.imgur.com/YRBGqUs.jpeg

    • reply
      July 19, 2024 7:52 AM

      Does the notification services for Evening Reading rely on Azure or Windows? Just received a bunch of them suddenly.

    • reply
      July 19, 2024 7:54 AM

      My wife is supposed to fly to Wales tomorrow. I wonder if that’ll happen.

    • reply
      July 19, 2024 7:57 AM

      Y2K24.

    • reply
      July 19, 2024 7:58 AM

      Microsoft Windows is so extremely bloated, nobody has an overview of the entire system and its external dependencies.

    • reply
      July 19, 2024 8:15 AM

      People blaming microsoft but this 100% on crowdstrike

      • reply
        July 19, 2024 8:23 AM

        This is what's confusing me. People are calling this a microsoft outage but they're just affected by the crowdstrike problem like everyone else?

      • reply
        July 19, 2024 8:28 AM

        There were two separate outages that happened almost simultaneously but had nothing to do with each other.

        Azure had a big outage and, three hours later, crowd strike affecting windows computers.

        • reply
          July 19, 2024 10:27 AM

          Ahhh nice lol

        • reply
          July 19, 2024 3:04 PM

          what if the azure outage was because of them testing the crowdstrike update/patch, and then azure got taken down so the folks doing the testing wasn't able to tell crowdstrike to "stop the release".

          • reply
            July 19, 2024 3:08 PM

            You see, the notification server for relaying the success of early tests had also crashed and by default, no error messages indicate a success!

      • reply
        July 19, 2024 8:50 AM

        Yeah, most media doesn’t understand

    • reply
      July 19, 2024 8:15 AM

      I'm completely dead in the water for work. All of the network traffic (even non-intranet) on our work laptops is routed through VPN for security reasons, and the VPN is offline still.

      Mobile Outlook/Teams works, but two years ago our work required us to install remote management software on our phones to continue using mobile Outlook/Teams, and I told them they can either give me a work phone or I just won't have mobile access. You can guess which direction they went, lol. So now my only means of updates is via text from my boss.

      And yeah, can't access literally anything on my work laptop since it's all routed via VPN. So just sit and wait today (or sit and play games while I wait lol).

      • reply
        July 19, 2024 8:24 AM

        open up Skyrim, that's your office for the day.

      • reply
        July 19, 2024 8:28 AM

        They asked you to install remote software on your personal phone? LOL

        • reply
          July 19, 2024 8:32 AM

          Yup, lol.

          My position was that I had mobile Outlook/Teams installed on my personal device as a convenience to the company. Their reasoning was that if the phone is lost, they want to have the ability to remote wipe it to protect company data. But I don't want some idiot screwing up and pushing the wrong button and wiping my personal device. Also, if they determine it's critical that I have remote availability, they can provide me with a work device.

          So I told them they have my phone number and can call or text if they need something urgent, and it's been like that the last two years. It's definitely a hit to the company because it used to be sometimes I'd see an email or teams message after hours and if I wasn't doing anything I might hop on to respond or resolve it. But now it just waits until I'm back the next day. /shrug

          • reply
            July 19, 2024 8:42 AM

            It’s funny that they can afford crowdstrike, the most expensive EDR but can’t provide a company phone for you.

            • reply
              July 19, 2024 8:55 AM

              Honestly, in the almost 6 years I’ve been here, I’ve had one instance of an unscheduled, off-hours emergency.

              Although we’re a big company and have some divisions that run 24/7, weeks have a pretty good culture from the top down that most things can wait until regular business hours. Unless it’s something that impacts production, it can usually wait.

              Plus, I’m in the software development division. I’m sure our tech ops guys have been up all night dealing with this but I don’t even find out we were impacted until I tried starting my work day.

          • reply
            July 19, 2024 8:56 AM

            What's so dumb is they don't need any sort of software for this. They can already kill access to Outlook/Teams if your phone is lost or if you quit. It only affects the apps themselves because it's just an account you have access to and they simply turn it off. No software is needed.

            • reply
              July 19, 2024 8:57 AM

              Yeah, our director of security is a bit of a fanatic. When I talked to him about it, he said users could download or screenshot things to their phone so they want to be able to do a full wipe. That’s when I said they can kick sand, lol.

              • reply
                July 19, 2024 8:59 AM

                He’s not wrong. But yeah you made the right call.

                • reply
                  July 19, 2024 9:00 AM

                  Yeah, if they wanted to provide a work phone they can out whatever they want on it.

                  But no way I’m going to let them have the ability to wipe my personal phone. I’m surprised how many people agreed to it.

              • reply
                July 19, 2024 8:59 AM

                Yeah my current company doesn't allow screenshots or copy/paste in Slack on iOS. I told security you know someone can just take a picture with any other camera they have trivially and you could see the look on their faces as if I blew their minds that preventing a screenshot is just theater.

                • reply
                  July 19, 2024 9:04 AM

                  Most security is just theater and redirecting liability. Same with financial audits. It’s all about optics and the ability to blame something/someone else when things go wrong.

              • reply
                July 19, 2024 9:38 AM

                director of security doesn't know how icloud works, amazing

            • reply
              July 19, 2024 9:08 AM

              its very normal to have some sort of controls on device that have access to work resources. in this case everything is working as intended, they should either provide device if the person needs to work off hours or not have that expectation. but the expectation that even a BYOD has controls on it for work resource is not odd

              • reply
                July 19, 2024 9:49 AM

                Controls within the apps themselves are fine. They don't need full control to wipe my phone if it's BYOD. As I said they can immediately block access to Slack, Teams, Outlook trivially. They do not need nor would I trust any company with that power over my personal phone as I have seen companies completely wipe a persons phone after they quit.

                If it's a company provided phone sure they can do what they want with it. My personal phone, you can lock down the individual accounts and that's all you get as you don’t need anything more.

                • kch legacy 10 years legacy 20 years
                  reply
                  July 19, 2024 12:33 PM

                  If your org uses O365 it is possible (easy even) for them to wipe your device even without company portal or whatever mdm they are using. Simply disabling the accounts should work in principal but in practice it doesn't work perfectly. Existing emails still remain on the device for example, and other work-related applications or data could remain.

                  The Company Portal creates a segmented area on your device for 'work' stuff. If you leave the company they can easily and effectively remove only the work apps and data.

                  It's really not as invasive as you think. They can see stuff like phone model, OS version, if it's jailbroken and so on. Doesn't show them the list of apps, at least how we set it up. Now, if your org's policy is to completely wipe the device upon leaving, that's not necessary and extreme IMO. Maybe if you're in some sensitive sector I guess, but seems extreme.

                  Source: IT guy who is working on this exact thing and completely wiped test phones that had no MDM or other connection to the org except for signing in to email.

      • reply
        July 19, 2024 8:37 AM

        Sounds like you've got the day off. Enjoy it.

        • reply
          July 19, 2024 8:47 AM

          Stuff’s slowly coming back online now. VPN is up, so that resolved a lot. But a lot of our app servers and databases are still down.

      • reply
        July 19, 2024 10:11 AM

        lol I had a bank try that. "Well your email is on there so no big deal". Ok, so my email is no longer on the phone.

    • reply
      July 19, 2024 8:31 AM

      What's really crazy is the post on /r/wallstreetbets before the update was pushed:

      https://www.reddit.com/r/wallstreetbets/s/Wy22QkZP0W

      • reply
        July 19, 2024 8:44 AM

        Nothing funny, WSB is full of wisdom no joke

      • reply
        July 19, 2024 8:52 AM

        That’s funny I just saw someone tweeted it was undervalued just now 😂

      • reply
        July 19, 2024 9:08 AM

        Wow.

        “CrowdStrike’s Falcon product contradicts their own guiding principle of “Zero-Trust Security”.”

      • reply
        July 19, 2024 9:38 AM

        Impressively accurate assessment.

      • reply
        July 19, 2024 3:23 PM

        Most of this is bullshit.

        • reply
          July 19, 2024 3:25 PM

          This guy has thing backwards. e.g. "Is CrowdStrike compatible with companies that run their IT systems on premises?" it was built for on-prem before Azure's cloud-based management of Windows, and thus part of the reason there was adoption was that you could deploy on prem extremely easily compared to the competition, in minutes, and manage everything from the CrowdStrike console, just as you can do today finally in say Azure now for Windows.

        • reply
          July 19, 2024 3:27 PM

          "they simply collect all of their customer’s data and display it on a dashboard." lol no

        • reply
          July 19, 2024 3:28 PM

          TLDR: see Google's acquisition of Wiz. If this guy were an expert the question should be "is CRWD suitable for cloud-based workloads?"

        • reply
          July 19, 2024 3:29 PM

          Ooh, the old bogeyman of conspiracy theories about the DNC hack: "CrowdStrike could potentially behave as a propaganda arm of the US government by creating “fake hacking stories” which are un-disprovable.They are able to do this due to information asymmetries in society."

        • reply
          July 19, 2024 3:30 PM

          "Corporations could buy CrowdStrike to spy on their own employees." this is a reason that they are more valuable to companies, not less, and thus the stock would be more valuable

        • reply
          July 19, 2024 3:32 PM

          "activity on the company issued laptop is reported to an internal dashboard which only an IT guy + a C-Suite admin have access to." also wrong, RBAC controls separate SOC operators who can see data from admins who can control deployment of the agent

    • reply
      July 19, 2024 8:31 AM

      My company seems unaffected but all our customers are, will be a quiet day.

      • reply
        July 19, 2024 8:35 AM

        Yeah same

      • reply
        July 19, 2024 10:30 AM

        We're unaffected and I thought all my side work clients were unaffected, but I have one that outsourced "security" to another firm, with me being the go-to for other stuff. Well the Internet has been down and it's not unheard of. But then the contact on-site asked if this was related to CS. I rapidly wrote no, but then thought better of it and had her contact the other IT firm to ask. Then I get a call from said firm confirming that yes, they use CS and that was most likely the reason the server was down.

        At first I was thinking, not my problem then, but these dudes are in Fort Collins and the office affected is on Castle Rock, where I live. So guess who felt morally obligated to take care of it.

        More like ClownStrike, amirite.

        • reply
          July 19, 2024 1:37 PM

          huh, the server resolved itself. maybe there is something to the "reboot a lot" fix? afternoon reclaimed!

    • reply
      July 19, 2024 8:33 AM

      What do you call a group of people who don’t want to work on Fridays?
      Crowdstrike

      Stolen from reddit, who may have stolen it from someplace else and posting it here so you can steal it and post it to your twitter

    • reply
      July 19, 2024 9:44 AM

      There goes my weekend. Thanks, Crowdstrike :(

    • reply
      July 19, 2024 9:54 AM

      I've never been more puckered to suddenly see "Your phone has been updated" as I was 5 minutes ago.

      • reply
        July 19, 2024 10:58 AM

        Lol.. same here. Our company does not use Cloudstrike at all, but it feels like bad juju to update or reboot anything at this point even though we aren't affected. It can wait until Monday. I was supposed to work on another project offsite today too... but fuck that. The ol' SysAdmin side of me is at the office just in case.

        Why? Because when you've been in the trenches.. the PTSD of reimaging or updating kicks in.

        *Stay strong my fallen IT brothers.. we feel your pain!*

        • reply
          July 19, 2024 10:59 AM

          *crowdstrike .. cloudstrike.. same thing*

    • reply
      July 19, 2024 10:09 AM

      You can't make this shit up:

      In 2010, McAfee released an update to their software that crashed WindowsXP systems worldwide, and required manual intervention to fix. https://www.zdnet.com/article/defective-mcafee-update-causes-worldwide-meltdown-of-xp-pcs/

      The CEO at the time was George Kurtz.

      The CEO of crowdstrike today is George Kurtz.

    • reply
      July 19, 2024 10:13 AM

      FML. Been up since 1 am fixing this shit.

      • reply
        July 19, 2024 10:27 AM

        Damn. They owe you a nice pizza party.

      • reply
        July 19, 2024 10:29 AM

        My buddy works for the county and has been at work since 10:00pm last night manually recovering workstations one by one.

        Sheriff dept, emergency services, county hospitals, etc were all offline last night. Crazy impact.

        He said he’s going to go into a coma after this because he’s been up for almost 30 hours straight. But on the flip side he’s hourly so making bank.

        • reply
          July 19, 2024 10:31 AM

          How do governments afford CrowdStrike? lol. I guess that’s where the property taxes go!

          • reply
            July 19, 2024 10:50 AM

            He's told me cyber security has been a massive focus for his county the last several years. Apparently there's been a few government agencies hit with ransomware that has cost them a ton of money, so other government administrations are scrambling to make sure they're secure. I'm guessing they see CrowdStrike as a cheaper alternative to paying a ransom or liability from leaked data?

      • reply
        July 19, 2024 3:15 PM

        We got it all done. I’m sure we’ll have a couple of stragglers Monday but we fixed 32 remote locations across the US, 4 manufacturing plants, and hundreds of remote sales people and couple hundred Windows servers. With just 5 people.

    • reply
      July 19, 2024 10:17 AM

      Hope you bois have backed up your bitlocker keys!

    • reply
      July 19, 2024 10:41 AM

      The fucking hubris of the tech world.

    • reply
      July 19, 2024 10:43 AM

      How does Crowdstrike just roll something out that bricks shit to EVERYONE. What the fuck is their process

      • reply
        July 19, 2024 10:46 AM

        at the very least i do not understand how it made it to so many clients. like there's no gradual rollout at all?

        • reply
          July 19, 2024 10:58 AM

          you can't really when it comes to security. Imagine if they did a gradual rollout with a working threat definition, your environment doesn't get the update in a timely fashion and now you're down.

          • reply
            July 19, 2024 11:05 AM

            yeah, i know that you don't want to do that with definitions, but you could still do a slightly gradual rollout to avoid situations like this. or at least if you're making changes significant enough to cause a BSOD you'd think that would be a separate gradual rollout. maybe it's something really dumb in the definitions themselves that is causing the bsod though, idk.

            regardless, it's probably worth slightly reduced security to avoid like, air traffic halting worldwide.

          • reply
            July 19, 2024 11:12 AM

            But like... a test on a Windows device would have shown that this did this? What did they test it on?

            • reply
              July 19, 2024 11:22 AM

              Windoze

            • reply
              July 19, 2024 11:22 AM

              Test?

            • reply
              July 19, 2024 11:27 AM

              When this was designed, because of the frequency of updates (more than one daily) it was assessed there was no time to create a test-before-deploy step

              • reply
                July 19, 2024 11:29 AM

                I guess one way to prevent malware is to brick the computer before the malware has a chance to do it...

              • reply
                July 19, 2024 11:36 AM

                "well we did one update today so the rest for today have to just go out full send worldwide without testing on any windows device whatsoever"

                It's not even like it's a hard to replicate bug! It fucking bricks windows from booting!

                • reply
                  July 19, 2024 11:50 AM

                  I haven't checked, does the crash only occur on boot, or does it occur immediately once the bad content is delivered?

                  • reply
                    July 19, 2024 5:01 PM

                    I'd have to know more about when the update got pushed to our systems to answer that. I know we got alerts about systems going down around 10:20PM last night. Not sure how that corresponds to the installation of the update(s) though.

            • reply
              July 19, 2024 11:28 AM

              Impact assessment:
              None to very little.

              • reply
                July 19, 2024 11:31 AM

                move fast and hopefully not break things

            • reply
              July 19, 2024 1:58 PM

              How do you know they didn’t test? It could be interpretation. Some automated test ran and they didn’t see a negative response (because the host was dead) so keep pushing.

          • reply
            July 19, 2024 12:15 PM

            This seems like a completely new build, not just an updated threat library or whatever.

          • reply
            July 19, 2024 3:07 PM

            in critical security patching, you actually *do* do a gradual rollout.

            think about google chrome, windows updates, apple updates, etc. in any critical security patching, it's actually way more important to slow down the update into cohorts.

            1) to make sure the update actually does what it's supposed to do
            2) don't take down your own servers while patching the world.

            the fact that there was no throttling means the folks at crowdstrike are rookies.

            • reply
              July 19, 2024 3:08 PM

              (and no matter how important you think crowdstrike updates are, no, they are not more important than *throttled* iphone updates that go to a much higher population than crowdstrike software)

      • reply
        July 19, 2024 10:53 AM

        Merge. Tests are green. Send it.

        • reply
          July 19, 2024 12:06 PM

          They forgot their final deployment guy is color blind. It was actually Red

      • reply
        July 19, 2024 10:57 AM

        And why does MS let 3rd parties push unsigned kernel modules to their users? Not that signing would have helped I guess :(

        But you'd think kernel modules would need some kind of approval process.

        • reply
          July 19, 2024 11:03 AM

          who said it was unsigned? they're signed. driver signing has been required for years

          • reply
            July 19, 2024 11:34 AM

            someone on the reddit secops thread said it was an unsigned driver, perhaps they were talking crap

            • reply
              July 19, 2024 11:36 AM

              It's a data file that the driver consumes. The assumption is that the driver code is hardened against bad data.

            • reply
              July 19, 2024 11:38 AM

              they're wrong in every sense that it's possible to be wrong. the driver *is* signed, and this *wasn't* a driver update. this was a data file update that the existing driver blows up when it reads

              • reply
                July 19, 2024 11:42 AM

                I think it's important to note that despite being data the control flow of the kernel can change based on this data in weird ways, and this is closer to a driver update than what most people might think

                • reply
                  July 19, 2024 11:49 AM

                  extreme case: you could imagine that a piece of data is so now big that it ends up accidentally allocated in a paged pool and then later, in a separate thread, minutes or ever hours later accessed in a non-paged context, and now there is a page fault in some OS code path owned by Microsoft that cannot handle page faults. This probably not what happened, but just an example of the complexity of data changes.

              • reply
                July 19, 2024 12:05 PM

                Maybe they meant the data wasn’t signed? Though it would have made no difference I guess

                • reply
                  July 19, 2024 12:07 PM

                  No, the data was not signed via the Microsoft-supplied path, but the data was supposed to be treated as malicious, as users can generate their own data

                  • reply
                    July 19, 2024 12:09 PM

                    users could suppress false positives by entering a list of paths in the cloud UI that the the driver should ignore, for example, and these paths eventually are served as data files

              • reply
                July 19, 2024 2:19 PM

                a data file that blows up a kernel driver is pretty bad.

                crowdstrike code is probably really bad.

          • reply
            July 19, 2024 1:02 PM

            It’s also enforced with Secure Boot too. Windows will straight out BSOD if it’s unsigned too when Secure Boot is active.

        • reply
          July 19, 2024 11:30 AM

          there is an approval process

          q: did you test it?
          a: yes
          q: is there bugs innit?
          a: no

          APPROVED FOR FRIDAY 12:01 AM RELEASE

        • reply
          July 19, 2024 11:34 AM

          This is not a new kernel module, and CrowdStrike's drivers are signed

      • reply
        July 19, 2024 11:03 AM

        He looked rough during his Today Show interview this morning. Kurtz drank water halfway through as if his voice was lost (yelling at his minions).

    • reply
      July 19, 2024 11:47 AM

      How is the media going to spin this by blaming Biden’s advanced age, all the while giving MAGA a pass?

      • reply
        July 19, 2024 12:22 PM

        It'll be Boeing's fault for cancelling the flights

        • reply
          July 19, 2024 12:24 PM

          Boeing forgot to tighten a thumbscrew on my PC, now it's BSODing!

      • reply
        July 19, 2024 12:31 PM

        It’s already on our Facebook HOA group. It’s a cyber attack and they are not telling the truth on what’s going on.

    • reply
      July 19, 2024 11:54 AM

      just finished fixing all my machines at work... GOD DAMN

      • reply
        July 19, 2024 12:22 PM

        Oof. Well done bud!

      • reply
        July 19, 2024 12:31 PM

        Nice! I have at least one friend working at a place where they were going to have to touch 10k machines. They’ll be working all weekend around the clock.

        • reply
          July 19, 2024 12:37 PM

          i'm in charge of about 100 Point of Sale devices and 100 computers and a few servers so definitely much more manageable. Point of Sale stuff was ez pz luckily but the computers took forever because of BitLocker.

        • reply
          July 19, 2024 3:03 PM

          Man, id quit. They can touch deez nuts

        • reply
          July 19, 2024 11:49 PM

          What’s he getting paid to do that?

    • reply
      July 19, 2024 12:26 PM

      Do you think this a good buying opportunity for the stock?

    • Zek legacy 10 years legacy 20 years
      reply
      July 19, 2024 12:26 PM

      I dunno, my PC is working fine.

    • reply
      July 19, 2024 12:29 PM

      I blame this on Onedrive!!

    • reply
      July 19, 2024 12:40 PM

      Thankfully our entire fleet of Windows Servers (~300) weren't affected. Maybe around 30-40?

      We were able to restore OS'es in about 5 hours between 2-3 of us.

      Apparently there are people with 1000s of machines affected, maybe even 10's of thousands, that are doing creative things like PXE booting and running embedded scripts to remove the files.

      Looks like I'll actually salvage a weekend!

    • reply
      July 19, 2024 12:47 PM

      As of the last update sent out around 1pm we are showing 68,000 of the average 83,000 devices online, still have hundreds of servers offline and thousands of clients lol

      • reply
        July 19, 2024 1:06 PM

        Yikes. Yeah we have an unknown number of clients affected. I just finished recovering our server infrastructure. Thankfully I'm not having to deal with the client-side.

    • reply
      July 19, 2024 1:12 PM

      Anyone else at an airport today? This sucks dick.

      • reply
        July 19, 2024 2:13 PM

        I flew into Chicago last night, whew!, hope things are all clear by Sunday

    • reply
      July 19, 2024 1:31 PM

      Anyone else working at an affected place? Are We Having Fun Yet???

      • reply
        July 19, 2024 1:42 PM

        Yeah, but I'm in charge of Linux systems so nothing I care about had issues. My auth / remote connections worked just fine. I was already logged into git so I was still able to make some merge requests and push things, but it was fixed like an hour or two later anyway so not a big deal.

      • reply
        July 19, 2024 2:42 PM

        We've been at it since 11pm PDT last night. Making steady progress. Our Windows admins are getting pretty adept with the Azure "mount the boot vol on a helper VM and edit, then reattach" dance.

        I'm a hybrid admin (mostly Linux) so the few places I have Windows admin privs, I fixed those servers in about an hour at 1am. Now I'm just facilitating.

        Our CrowdStrike sales rep sent $50 Uber Eats gift cards to each of us as apology / fuel. I almost wanted to reject it.

      • kch legacy 10 years legacy 20 years
        reply
        July 19, 2024 3:24 PM

        We didn't have it too bad honestly, most of our stuff recovered on their own (somehow). A couple servers required intervention. Handful of users had BSOD on their workstations but that's it. Most were off/sleep.

        Relatively small org.

    • reply
      July 19, 2024 1:48 PM

      I'm the Mac admin but I came into the office to drill home the superiority of macOS assist in manual PC remediations.

    • reply
      July 19, 2024 2:21 PM

      https://misskey.io/notes/9vw052ic52zv017x

      "Forced reboot almost destroyed my asshole"

    • reply
      July 19, 2024 2:48 PM

      Whoever is responsible for this colossal fuckup at CrowdStrike needs to put this on their resume, because it's extremely impressive just how badly they fucked up. Who else can say they brought down half of the worlds computer systems?

    • reply
      July 19, 2024 3:04 PM

      I wonder how many help desk / admins just straight up quit today

    • reply
      July 19, 2024 3:30 PM

      A deep dive of the root cause from some twitter dude:

      https://x.com/Perpetualmaniac/status/1814376668095754753

      "Crowdstrike Analysis:

      It was a NULL pointer from the memory unsafe C++ language."

    • reply
      July 19, 2024 5:09 PM

      I just got home after getting to work at 6:30am. Going back in as soon as I wake up tomorrow morning. Sigh...

      • reply
        July 19, 2024 5:10 PM

        :(

      • reply
        July 19, 2024 5:14 PM

        Are the turbolasers still down? Asking for some friends.

      • reply
        July 19, 2024 11:37 PM

        I just rebooked my flight to Japan. It was supposed to be this morning at like 8am. Direct flight with my wife. Even dropped another $600 for premium seating.

        Because of the crowdsuck crash they automatically put me on a flight tomorrow at 11pm with a 2 hour layover and coach seats, gave me a $600 coupon to use… eventually.

        But they got my wife on a flight today at noon, so she’s already in the air.

        We spent 4 hours in line at the airport to deal with rebooking. Airline only had 2 customer service reps.

        I won’t believe I’m on this flight until I’m actually in the seat.

    • reply
      July 19, 2024 5:21 PM

      Take a break

      https://i.imgur.com/cFWt2K6.jpeg

    • reply
      July 19, 2024 7:01 PM

      https://www.crowdstrike.com/blog/technical-details-on-todays-outage/

      Logic error when evaluating named pipes.

    • reply
      July 19, 2024 7:29 PM

      I wonder how many microsoft engineers were just tasked with making the BSOD say "_____ VENDOR FUCKED UP AND IT CRASHED"

    • reply
      July 20, 2024 12:24 AM

      (Video with sound)
      https://i.imgur.com/sRu0VW4.mp4

    • reply
      July 20, 2024 12:27 AM

      (Video with sound)
      https://i.imgur.com/xvQcXzH.mp4

Hello, Meet Lola