Pre-market, it got hit by like 14 points. Right now, it’s only losing 6. When the market opens, let’s see if they are putting in the past, or punish them.
1 Like
I think Crowdstrike will take the biggest hit - Microsoft will get hit a bit but to be fair to them they shouldn’t.
It’s Crowdstrike that appears to have released an update to such a critical system without testing it - and Microsoft can’t be blamed for the level of access Crowdstrike has to Windows as security systems like that require such permissions.
The fact the computers can be brought up in safe mode shows it’s not really Microsoft but a bad bit of software installed - however it does highlight that Microsoft probably do need to think of some way to allow IT teams to remotely start computers in safe mode and run updates, not that that’s an easy thing to do while keeping it safe from hackers…
If anything, it might allow Microsoft to continue with their push for software and patch certification at a hardware level - they started it a while back for Windows 11, and I can’t remember if they dropped most of it.
However if Microsoft certified the patch to go via Windows Update then they’d have a problem - in that case yeah they deserve to get a bit of a hammering alongside Crowdstrike and we can finally expect to see them increase their QA teams for such certification.
That may lead to increased costs for their partners though, which will be passed on to their customers and onto us via their price hikes…
6 Likes
Someone mentioned on Mastodon running the update first on a separate pc/sever done for testing to make sure it didn’t break anything. But I can see that being a issue for companies that don’t want to maintain extra hardware.
1 Like
Should have been done at CrowdStrike as part of the standard software development lifecycle.
7 Likes
Looks like “apology accepted” went through as the damage on stock went from -14 points to 3 points. If it happened during the opening hour, I think it would have been fatal.
While I don’t work for Crowdstrike, I work for one of their largest competitors. The amount of marketing materials being generated to take them out of some of the world’s largest clients is massive. BSOD’ing the heavyweights of the world is more damaging than most of the largest cyberattacks… ever.
What an irresponsible testing/Q&A lifecycle. My company’s blog post is 25% lol Crowdstrike, and 75% “we don’t suck like that.” What a crazy world. It’s not a MS problem, and that’s why the stocks have mostly rebounded (as you pointed out), but Crowdstrike was bleeding today.
4 Likes
CS stock going down 14% is such a steal for the large investment firms - they are beyond excited to get in on at a low point for a company on a seemingly infinite growth loop. I bought more CS stock this morning, too!
4 Likes
Going Live
3 Likes
Yep I imagine their rivals (such as your employer) are having field days with this.
It just beggars belief that this got through - given it seems to be affecting VMs, my best guess is that they did their development and testing on non-VMs and didn’t have testing on VMs as part of their regression tests.
Given that cloud security is such a critical part of their business though, that is just wild…
(I’d hate to be the developer or tester involved - that stain is never coming off your CV lol)
1 Like
It sounds like their Dev and Q&A cycles don’t incorporate live Windows environments or employ static/dynamic code analysis (or app analysis) after passing their dev sandbox. It’s a small, expensive, vital corner to not cut - but when you’re the only real game in town for what it is they do best; you get away with it. Until you BSOD everyone, then you don’t.
My company has completely reversed course and now won’t publish any material re: #clownstrike and now we’re just offering to support effected customers with our patch/patchrollback/software rollback toolset. All of these companies share the top end talent and execs, so slinging mud in the public arena is ultimately going to drown the mudslinger first.
That Q&A manager is going to be persona-non-grata for the rest of their career. They had to ultimately greenlight the package and roll it out? Yiiiiiiiiikes!
2 Likes
Yep the sympathetic approach and offering to help are golden techniques - given Crowdstrike is likely to be unreachable today and their customers will be looking around for help and to vent.
Agreed they’ve definitely cut some corners - the “outside of page file range” error sounds like they’ve been doing low-level code and designed it purely for physical drives and forgotten that on a shared drive / SAN for VMs those page file ranges could be very different.
It’s one of those where even though we’ve got all the clever stuff nowadays around code analysis, automated regression tests and the ability to spin up extra environments for particular tests, the most basic of tests would have found it.
It’s such old school testing - if you’re rolling out something, make sure it works OK on a few of the different configurations your customers use, it’s the bare minimum that goes back to the days of web development meaning you’d test it in IE, Firefox, Safari and Chrome - or testing games with both AMD and Nvidia graphics cards.
It’s just such a basic thing to get wrong - while the more advanced stuff they could say “we have an upgrade project planned that was bringing all of that in”, this stuff is just things they should have been doing since the very start a decade or so ago.
I imagine the QA manager / tester / devs may say “I left just before that happened” when asked about their CV in future lol - I imagine their shareholders are going to be absolutely livid though if this does have some long term financial implications for the firm (which it will - even if just in incentives to clients to stick with them when the contract is up for renewal)…
2 Likes
I do wonder if the marketing budgets are divvied up by revenue - so the Europe, Middle East and Asia market is likely not one of Xbox’s biggest (outside maybe the UK) so doesn’t get as much of a cut.
I’m hopeful the big marketing budgets of CoD will help by marketing Xbox by association - as ABK did spend big on marketing which Xbox themselves generally haven’t so much (can’t blame them really - to match Sony or Nintendo’s marketing while still not having much chance of really reducing their market share, you’d have to spend a much larger portion of your gaming revenue as you’re in third place).
So hopefully the games marketing from CoD etc. will help to close that spend gap…
CrowdStrike can eat my whole ass. Spent the whole morning bringing back all of our servers.
8 Likes
Wild to see a person started his first day at Crowdstrike and now released all on one day.
You have all of my sympathies. Today’s the first day in six months that my clients aren’t yelling at me for something, which means everybody is recovering from CrowdStrike’s nonsense. So selfishly I’m enjoying not being on fire, but I also know that it’s not a good reprieve, it’s the tide pulling back before the tsunami
3 Likes
We just switched to crowdstrike last year, now we have a post-mortem scheduled for monday to decide whether to stick with it or not. Or set our sensor upgrades even further behind.
1 Like
Oh those calls are so much fun. You must be excited! /s
If I were still in new-business sales, I’d be rubbing my hands together like Phil over an acquisition proposal at the thought of how many companies are going to inquire into CrowdStrike alternatives. Being on the client retention and expansion side of the house now, it’s just reassuring everyone that they’re okay - and no, I can’t really help them unless they want to tear CS out completely.
2 Likes
i dont know what you expect if you install a software from checks notes crowd strike on your server. its in the name 
2 Likes
Woke up. Read about Crowdstrike situation. Started connecting to work VPN.
It’s down.
Message Senior on Discord. It’s not CrowdStrike. Still he has to head to office and restart it from there.