Florida schools plan to vastly expand use of AI that mistook clarinet for gun

Human review didn't stop AI from triggering lockdown at panicked middle school.

See full article...

 

[ranthog]

Going by the news we see from America, a finger gun (or sandwich gun, for that matter) will at most get the kid a 3-day suspension.

Unless of course some idiot in authority involves the police -- in which case there may not be be a swat team, "just" one or two officers who may decide the appropriate response is to arrest and handcuff the child, and take them down to the station for their parents to rescue. Especially if the child is non-"white".

That assumes they don't just shoot the child.

 

[numerobis]

That assumes they don't just shoot the child.

Or the two kids behind the finger-gun kid.

 

[Bernardo Verda]

Yeah there's no split-second decision making that should be happening here. Even if they called the cops after a split second, they don't teleport into the school -- there's no reason they can't stay on the phone with 911 while they do a further review so that by the time the cops arrive they are aware that it was a musical instrument. That doesn't solve the problem of the lockdown (which would be triggered immediately) but at the very least they have the ability to prevent it from escalating!

If there really is are retired cops or soldiers, actually reviewing the flagged images, and they really are actually making split-second decisions, then obviously they don't have enough people in the loop, and the expert reviewers are drowning in a deluge of images (which would be both inherently stressful and intrinsically error prone).

 

[KoshcheiTheDeathless]

If it misidentified a clarinet as a gun due to how it was held, I wonder if it would misidentify a gun as not a gun if it was being held in a "non-gun like" way.

I guess that this scuttles the school orchestra's plans to introduce a leadwinds section.

 

[C.M. Allen]

If there really is are retired cops or soldiers, actually reviewing the flagged images, and they really are actually making split-second decisions, then obviously they don't have enough people in the loop, and the expert reviewers are drowning in a deluge of images (which would be both inherently stressful and intrinsically error prone).

That presumes that there are people in the loop to make those calls. The results of this system 'operating correctly' strongly suggests that's not the case. Otherwise we wouldn't be hearing about any of this. So it would appear to be completely or almost completely automated. Which means officers called to the scene of a 'school shooter' by this system aren't going to 'know' anything more than that. And that means they won't know how to respond correctly. Either they won't take it seriously (because the error rate is upwards of 90% or worse) when it actually matters, or they will take it seriously all the time and eventually shoot and kill innocent kids. In either case, this system 'working correctly' isn't just putting lives in danger, but in more danger than the problem it's being sold as a solution to. It's not unlike saying the 'solution' to drunk drivers is to have them fly helicopters home instead.

 

[ZAP!!]

A whole new market is now open because the USA won't do otherwise to stop gun violence. Instead of spending money on something that doesn't really work, it would cost nothing to implement better gun laws. No parent in their right mind would want their child to experience a false positive.

 

[graylshaped]

“We don’t think we made an error, nor does the school,” Alaimo said. “That was better to dispatch [police] than not dispatch.”
...
And seemingly rather than probe why the images weren’t more carefully reviewed to prevent a false alarm on campus, the school appeared to agree with ZeroEyes and blame the student.

“We did not make an error, and the school was pleased with the detection and their response,” ZeroEyes’ spokesperson said.

I don't know the details of the police response, since the Wapo article is paywalled, but I would take issue with any response that included officers swarming with guns drawn without the opportunity for an independent, clear-eyed, non-ego driven individual with an agenda to review the video of the student. If my kid's school was pleased with the detection of a clarinet, I would be That Guy at the next school board meeting.

This was an error that endangered students and there should be a transparent discussion over what specific steps ZeroEyes will take to mitigate against a repeat.

 

[recalcitrant]

I was so relieved the school and AI company were able to blame a school kid in band practice holding a clarinet as the problem rather than:

• The AI system closely monitoring them all
• The company providing the AI services
• The lack of ANY human verification, even if while the police are on route,
• The school for implementing unreliable systems on a hair trigger
• Or perhaps gun policy that creates this entire mess

AI has now reached the era of victim blaming because it's more important for a company to maximise profits. There will be plenty more of this to come.

 

[calvinist]

CEO: "This is an incredible product, but what should we call it? It should evoke a sense of vigilance, but also quite confidence and safety!"
sarcastic programmer: "How about ZeroEyes?"
CEO: "That's PERFECT!!"
sarcastic programmer: "uhhmmmmm . . . ., alrighty then."

 

[Hmnhntr]

People in the US will literally do anything other than actually enforce gun laws and regulate deadly weapons. This is a system bending over backwards to avoid telling lunatics they should not have access to deadly weapons.

It’s just insanity.

But if I can't own a deadly weapon, how can I fantasize about the opportunity to legally kill someone? And if I don't do that, how can I get hard?????

 

[Hmnhntr]

To add, though coverage wouldn't be great, for the same costs Kansas is looking at undertaking, that would pay roughly one school resource officer per 10 schools.

As sort of noted towards the end of the article, in general and at best, this maybe gives first responders, school staff, etc. seconds of extra notice.

Telling is the mention that they detected over a 1,000 weapons in 3 years of operation! They then make note that they have facilitated 10 arrests.

Yeah. So how many of those were real weapons? How many constituted true threats (and weren't some elementary schooler that brought a swiss army knife to school, etc.)? How many wouldn't have been spotted by staff, SROs, or students and reported and resolved anyway?

I am all about keeping kids safer, but to a degree, having more officers available likely has significantly more tangible (good) impact on school safety. Of course, that too has negatives when SROs are used to enforce school discipline.

Doesn't this (bolded section) indicate a false positive rate around 99%?

 

[Hmnhntr]

Cool story, bro.

An "eighth of an inch of a gun" could be literally anything. Any 1/8" black bar could be "a gun". Insanity that they would even claim something that asinine. Wouldn't you want to error on the other side? You know, being more sure that there actually is a gun before calling cops?

 

[Hmnhntr]

I do have kids, and this incident is nuts—I hate it. One of mine plays the oboe, so it hits home. But I hate this for all kids and their parents.

And I agree with you in principle that streamed “security” video is generally objectionable, though also generally legal.

But no one is super-excited about accessing that video to watch my kid pick his nose as he walks down an empty school hallway. IMHO, “live video of my kid” in school isn’t the threat you imply it to be. I’m picking up a whiff of “won’t someone think of the children?!?” in your comment, though maybe that’s far from your intent.

The threat here—to me anyway—is primarily the danger of AI auto-swatting middle schools, and secondarily the erosion of norms around privacy and reasonable expectations of privacy.

As scary as it sounds that “people” could hack in and watch one individual kid—and they could!—we as a society have much bigger fish to fry.

But why are we surveilling kids at all in the first place? This isn't a "think of the children" thing, it's a refutation of the "think of the children" BS that led to these cameras being installed in the first place!

Surveillance should be utilized only when necessary and effective. And twice as much caution should be applied when surveilling vulnerable groups of people, like children. Especially since we have already seen cases of people using these sorts of systems for their personal ends.

There's always a bigger fish. That's beyond a shitty excuse to ignore something. Do you not do your chores because there are bills to pay?

 

[Hmnhntr]

When I was in middle school we were able to come and go from school grounds at will if we weren't in class.
One generation after mine, they weren't allowed to leave.
Two generations after mine, they became used to going through metal detectors to enter the school.
Three generations after mine, they have constant video surveillance and SWAT teams descending on schools willy-nilly.

Fear and lack of agency are what's being taught.

I've had people tell me my parents abused me because they let me walk home a little over a mile back from school. People don't even think it's safe to trick-or-treat anymore. Security theatre doesn't make anyone safer, and it certainly doesn't make people feel safer. It just puts the shoe in the door for the idea that danger is everywhere, and the only solution is constant vigilance, at the cost of anything else.

 

[mikeschr]

I've had people tell me my parents abused me because they let me walk home a little over a mile back from school. People don't even think it's safe to trick-or-treat anymore. Security theatre doesn't make anyone safer, and it certainly doesn't make people feel safer. It just puts the shoe in the door for the idea that danger is everywhere, and the only solution is constant vigilance, at the cost of anything else.

I'm going to guess that the people who told you you'd been abused went to school in the '80s or later and think of overprotection and helicopter parenting as normal. This is all so unhealthy and dangerous.

 

[Kenjitsuka]

“That was better to dispatch [police] than not dispatch.”

If it's better, will you foot the bill? Oh, right, zero eyes, zero responsibility, 300% more cash!

 

[Tofystedeth]

Hi, clarinetist here. We quite often carry our licorice stick in that alleged "looks like rifle carry position" . Just as with a rifle, or a stick, it's a safe way to transport it.

Huh, I figured you for a cellist.

 

[ranthog]

I was so relieved the school and AI company were able to blame a school kid in band practice holding a clarinet as the problem rather than:

• The AI system closely monitoring them all
• The company providing the AI services
• The lack of ANY human verification, even if while the police are on route,
• The school for implementing unreliable systems on a hair trigger
• Or perhaps gun policy that creates this entire mess

AI has now reached the era of victim blaming because it's more important for a company to maximise profits. There will be plenty more of this to come.

After hearing that response, I know when I was in school I know pretty much the entire nerdy good kid crowd would probably be trying to figure out how to cause the AI company's system to flag things that don't look like guns as guns. Proving the school administration stupid was a fun game, and it's not like the school can actually give someone consequences because their AI is stupid.

You just don't prove yourself to be this fucking stupid like this to teens. Nor do you talk down to them and pretend they are going to accept this is somehow the student's fault.

I know it was never anything this high stakes, but dear god did we harass the administration in ways they couldn't do anything about.

 

[numerobis]

You just don't prove yourself to be this fucking stupid like this to teens

Reality says that actually, people with power do it with stunning regularity.

 

[gnasher729]

Is there something in the Florida tap water that changes every male adult to “Florida man?”

Yes, and that includes all adult transgender males aka “transgender women” in the civilised world.

 

[gnasher729]

But if I can't own a deadly weapon, how can I fantasize about the opportunity to legally kill someone? And if I don't do that, how can I get hard?????

Charlie Kirk really supported the idea that everyone should at least have the tools to kill other people. He didn’t quite appreciate the idea that someone might see him as “other people”.

 

[gnasher729]

Doesn't this (bolded section) indicate a false positive rate around 99%?

Sure. But guns are rare. So if people with guns and people without guns are equally likely to be misjudged, then 99% of all errors happen to people without guns who become false positives, and only 1% happen to people with guns who become false negatives.

That happens in medical exams, where failing a test for a very rare disease is most likely a false negative and not a correct positive and not a reason to panic.

 

[Dean C. Rowan]

I feel like I should chime in here. This is my nephew’s school, he’s in the band there and knows the “shooter” rather well. Take it for what you will, but the clarinetist in question is doing fine, hasn’t felt like anything is his fault, and has said he’s rather enjoying this moment in the limelight even though it’s for a stupid reason. At 13 years old, they’re not considering the ramifications us old people are and they’re just psyched as hell that they have “I was there!” stories for their TikTok followers.

Bravo to the reed player (Benny Goodman in camo) and to your nephew. Their take is exactly right. But I disagree that they're "not considering" important factors but instead glomming onto prospects of celebrity. They're actually assessing the situation for what it is. Kids are smart. (Of course they are! We raised them!) I get queasy when I find myself condescending too much to them.

 

[Pooga]

Until some trigger-happy cop decides that a cellphone, or a tape dispenser, or a stapler, or whatever is 'gun-like' enough to open fire. Because, you know, that's never happened before. Law enforcement is the beacon of restraint, civility, and thoughtful, careful actions....

Well, @el_oscuro did say it was their "teenage version of themselves" that would jump all over it. Teenagers are well known for doing stupid and dangerous shit...

 

[thelinuxgamer80]

You do not even need the ai wired to the kill button to result in a kid getting killed. We have examples of people prank calling swat on another person and it ending in violence. This should not be anywhere near our schools.

 

[mexaly]

How fortunate that their AI system doesn't wield its own weapon.

 

[numerobis]

How fortunate that their AI system doesn't wield its own weapon.

Yet.

 

[Bernardo Verda]

Yet.

"You have 20 seconds to comply."

 

[gnasher729]

You do not even need the ai wired to the kill button to result in a kid getting killed. We have examples of people prank calling swat on another person and it ending in violence. This should not be anywhere near our schools.

If you have a law-abiding citizen who has a loaded gun to protect himself from burglars, and armed police who have been told there’s a dangerous shooter with hostages inside, then violence is no surprise.

 

[shodanbo]

So, if I'm reading this right, they're telling the student it's his fault in that he asked for it by the way he dressed and the way he held himself?

Wouldn't be the first time a tech company blames us for holding it wrong.

 

[jonah]

If the clarinet player had been brown or black, he could have been shot.

These systems are a waste of money at best, and actively endangering students while providing absolutely nothing of value in return at worst.

So yes, let's absolutely implement them everywhere.

Fucking idiots.

This fall they put metal detectors at my kids' high school. I asked the principal how many incidents they'd had, in say, the past three years of weapons being confirmed on school premises. He evaded the question. I suspect because he didn't know.

Then I found out there was an incident last year where a student was stabbed at a different high school in the same district. A single incident. Resulting in literally millions of dollars spent on stupid metal detectors THAT DON'T EVEN WORK.

 

[dzid]

If the clarinet player had been brown or black, he could have been shot.

These systems are a waste of money at best, and actively endangering students while providing absolutely nothing of value in return at worst.

So yes, let's absolutely implement them everywhere.

Fucking idiots.

This fall they put metal detectors at my kids' high school. I asked the principal how many incidents they'd had, in say, the past three years of weapons being confirmed on school premises. He evaded the question. I suspect because he didn't know.

Then I found out there was an incident last year where a student was stabbed at a different high school in the same district. A single incident. Resulting in literally millions of dollars spent on stupid metal detectors THAT DON'T EVEN WORK.

Of course. I've had several Black friends killed by cops. As far as the AI or automated surveillance, if you were to ask the community-oriented police or hell, even the gang unit cops (the ones that don't have incentive to push these systems) they'll tell you it just gets in the way. That's the reality.

 

[wk_]

I doubt there's actual human review in the loop for this service. If there were, then the delay caused by waiting for human review to confirm a gun is present would eliminate the entire theoretical speed gained from using Big Brother AI to call the police before anyone calls 911.

It would be beneficial, if alarm is raised, and SWAT is on the way, someone to look at the actual video and to get as many information as possible. Sending the SWAT team into (what your system believes to be) a dangerous situation and/or knowing that the kids are in danger and not trying to get a closer look and leaving to AI to handle it is simply irresponsible.

 

[dzid]

leaving to AI to handle it is simply irresponsible.

'despicable' is the word I'd use. Humans need to be in control of this and those people have to care.

 

[Zeppos]

People in the US will literally do anything other than actually enforce gun laws and regulate deadly weapons. This is a system bending over backwards to avoid telling lunatics they should not have access to deadly weapons.

It’s just insanity.

Teach in a high school in the EU. No shootings (yet) like in the US. But a lot of scared kids in the classroom when kids were popping balloons nearby. Kid had a box cutter. Caused a small stampede of kids fleeing the scene, while the kid with the cutter clearly misjudged the situation but was rather innocent.

Only a matter of time, I am afraid, before someone realizes that this is a way to get a lot of attention. Let's hope they can't find guns on the illegal market. But with the right connections, that should not be that hard.

 

[SixDegrees]

Teach in a high school in the EU. No shootings (yet) like in the US. But a lot of scared kids in the classroom when kids were popping balloons nearby. Kid had a box cutter. Caused a small stampede of kids fleeing the scene, while the kid with the cutter clearly misjudged the situation but was rather innocent.

Only a matter of time, I am afraid, before someone realizes that this is a way to get a lot of attention. Let's hope they can't find guns on the illegal market. But with the right connections, that should not be that hard.

I don't think they need to rely on illegal guns, depending on what part of the EU you're talking about. Guns are legal to own in many EU countries - better regulated than in the US, but still reasonably available.

 

[cygam00]

Remember when people used to react to car alarms? Now people just hate the dumbasses who still use them.

AI "security" - pretty much going to end up the same way.

 

[dzid]

Teach in a high school in the EU. No shootings (yet) like in the US. But a lot of scared kids in the classroom when kids were popping balloons nearby. Kid had a box cutter. Caused a small stampede of kids fleeing the scene, while the kid with the cutter clearly misjudged the situation but was rather innocent.

Only a matter of time, I am afraid, before someone realizes that this is a way to get a lot of attention. Let's hope they can't find guns on the illegal market. But with the right connections, that should not be that hard.

I have family in the EU, and their kids didn't seem to have issues. There's always the possibility of tragedy. I've seen so many kids die personally, as young as thirteen, and it's always tragic. It's almost always preventable.

ETA: Very, very importantly, no automated surveillance system would have done shit to prevent the deaths I've seen or the deaths of those I knew in the neighborhood. That requires humans. Humans that care.

 

[innov9090]

What the hell is this Luddite comment section on? It was the superintendent's JOB to review the images. They did not review the images and just called the police straight away.

How is this in any way the system's fault, where it presented the images for human review? Any tool used incorrectly will cause problems.

I swear 99% of Arsians are just busy frothing at the mouth against AI instead of thinking critically.

Literally all alerting systems in the world require human triage. Like JFC if people are this incapable of critical thinking, maybe we humans are all just better off replaced with AI.

 

[dzid]

What the hell is this Luddite comment section on? It was the superintendent's JOB to review the images. They did not review the images and just called the police straight away.

How is this in any way the system's fault, where it presented the images for human review? Any tool used incorrectly will cause problems.

I swear 99% of Arsians are just busy frothing at the mouth against AI instead of thinking critically.

Like JFC if people are this disgustingly incapable of critical thinking, maybe we humans are all just better off replaced with AI.

The point is that the system has no value. The details matter.

ETA: In my opinion, you are coming at this issue from the perspective of someone who does not understand inherent bias in our law enforcement system itself.