RE:RE:RE:RE:RE:The Competition…Evolv Is Not A "Weapons Detector", It Is a Smarter Metal Detector By Nikita Ermolaev, John Honovich, and Donald Maye, Published Jan 11, 2022, 09:44am EST | Email This While Evolv's CEO declared Evolv is "a weapons detector, not a metal detector", IPVM has determined that Evolv is a metal detector, though smarter than conventional ones, based on conversations with Evolv's executive team plus research into Evolv's patents and academic papers. IPVM Image Evolv cannot precisely detect what is truly a weapon, regularly mistaking what they admit are "benign" objects, including Chromebooks (as we recently analyzed) plus 3 other common false alert sources, as weapons. In this note, we examine Evolv's technology, including how it operates, the systematic false positives it generates (from its own documentation), and how it is clearly smarter than conventional metal detectors though still limited in its intelligence about "weapons". Not A Metal Detector Evolv declares, without qualification, that they are "not a metal detector", in their FAQ, as excerpted below: IPVM Image False Positives Theme - Long Metalic Objects However, Evolv's most common causes of false alerts include umbrellas, strollers, eyeglass cases, and laptops, as the excerpt from the company's 2021 solution brief below shows: IPVM Image They share a common pattern - objects that are metallic and long, as the illustration below visualizes: IPVM Image Evolv cannot adequately distinguish between an actual weapon - like a gun - and objects that are clearly not weapons (eyeglass cases, laptops, strollers, umbrellas) because those objects have rough characteristics that both groups share and Evolv does not have sufficient intelligence to differentiate between them. Conventional Vs Evolv Both Evolv Express and Conventional Metal Detector are active magnetic detection systems, however, Evolv Express uses a more sophisticated analysis of magnetic response data. While conventional metal detectors simply analyze the response from the metallic object, Evolv Express includes a substantial library of pre-determined threats, measuring the polarizability index of metallic objects being scanned and a reconstruction module. Evolv uses a reconstruction module comparing the response from the metallic object scanned to a model they calculate. From a 2018 Evolv patent describing this: The reconstruction module ... can be configured to determine a best-fit object or objects which can be defined by a set of attributes including position, speed, time-offset, and polarizability index. Best fit can be determined by a cost function that measures the difference between the actual measurements and those predicted by a model... The patent further explains that once the best-fit object is found, a corresponding value of the Magnetic Polarizability Index for the object is calculated and passed on. This index characterizes the object's magnetic response and is dependent on the shape, size, and electric and magnetic properties of metal. The passed value is compared to a library of predetermined threat signatures to make a threat/non-threat decision. Additionally Evolv Express has machine learning capabilities that can help with threat/non-threat discrimination, from the same patent: …a threat/non-threat decision can use a classifier trained by in a machine learning process, in which many labeled examples of threats and non-threats are used to train the magnetic sensing algorithm to determine if a newly detected object should be characterized as a threat or non-threat. Conventional Metal detectors use less-sophisticated data for object characterization (the delay time, or phase and amplitude of a received signal) and do not reconstruct the response. However, even though Evolv is smarter than a conventional metal detector, its process still has fundamental technological limitations in differentiating benign objects from actual weapons. Evolv uses a statistical model to assess how likely a scanned object is to match a library of known weapons. This is based on certain magnetic properties of these weapons (such as the polarizability index). Nonetheless, various benign objects (such as umbrellas, strollers, laptops, etc.) have similar properties to the library that Evolv uses. Of course, all of this assumes the weapon is metallic as Evolv's process is limited to analyzing metallic objects and therefore cannot analyze non-metallic weapons. Value of Smarter Metal Detectors Making metal detectors smarter does provide potential benefits, e.g., reducing the number and types of metallic objects that generate false alerts. But Evolv, like its conventional competitors, detects metal and still has fundamental limitations in detecting certain benign metallic objects and missing non-metallic weapons. Understanding this is important to properly evaluate both the benefits that Evolv can deliver and what limitations or risks are present with Evolv's system. Evolv Feedback / Now Refuses While Evolv is now refusing to speak to us (since the end of December), we did speak with Evolv's executives earlier in December, where the company acknowledged that they only detect metal objects and that they often generate false alerted on non-weapons. We even asked Evolv about our assessment is that Evolv is actually just a smarter or better metal detector. At that time, they asked to take more time to think about that and, thereafter, they refused to speak with IPVM. As we do with all subjects we report on, if or when Evolv wants to provide feedback or input, we will immediately update our reporting and share it with the public. Evolv has refused to speak with IPVM and we will not be contacting them for comment going forward, unless and until they contact us saying they are open to speaking with us. I am making this public so that there is no confusion or unclarity about what is happening and in case someone at Evolv sees this and intends to do otherwise. As a public relations strategy, I think it is a very foolish one. Even Hikvision management periodically communicates with us, and that's after our series of investigations uncovering human rights abuses and PRC government control. Our goal in speaking to subjects is to help them share their perspective, we can always gather extensive evidence from the company's marketing material, public financial filings, FOIA requests, etc. for strong foundations of our reporting. On January 3rd, I sent the following email to Evolv's CEO Peter George: Peter, My name is John Honovich and I am the CEO of IPVM. Our organization has tried, for months, in good faith, to speak with Evolv. From Dana's [Evolv CMO] response and then further non-response, our understanding is that Evolv intends not to talk to IPVM going forward. If this is incorrect, please let me know by the end of tomorrow, Jan 4th. Because of Evolv's decision, IPVM will not reach out to Evolv with questions unless and until Evolv reaches out to IPVM first to inform us that Evolv is willing to talk. We are always happy to talk and incorporate feedback from subjects of our reporting but if a company makes it clear that they have no intention of doing so, we are not going to bother them. Prior to that, on December 28, Evolv's CMO Dana Loof emailed us: Over the past few weeks, it’s become evident that Evolv and IPVM have different perspectives regarding the best way to share information to improve public safety. We have concluded that it is not in the best interest of our customers, the security profession, the public, or Evolv to support IPVM’s efforts at this time. As we move into 2022, we remain committed to our founding mission: to make places where people work, live, and play safer. While we have no idea what they mean by "support" since we do not ask companies to "support" us, based on multiple emails from IPVM back to Evolv asking if this means they will not speak with us, with no response from Evolv, their point is clear. Evolv is refusing to speak to IPVM. The best backgrounder is here: Evolv Objects To IPVM Reporting on Its Weapon Detection Weaknesses. Any questions or comments, please let us know. And we are still happy to speak to Evolv at any time, if they change their mind and let us know. I believe the 'shame' is that Evolv is hiding this information from the general consumer, and putting their trust in sales associates to disclose these weaknesses instead of putting them in writing as a company. I can try to illustrate the risk with my experience: We are currently evaluating this product. The two IPVM reports exposed weaknesses that we did not know about because our Evolv salesperson did not disclose them to us. I cannot say for certain they would have never disclosed that info, but how far down the sales process do we have to be to get that information? By the time a large organization like ours (or anyone large enough to want to buy something that scans hundreds of people) learns about these weaknesses, their might already be 'skin in the game'. Evolv knows this and uses that to pressure a buy, after its too late to pull out. And that's simply one bad scenario. The morality of hiding your products weaknesses behind a chain of trust and a salesman does not look good. What else could Evolv be choosing to not disclose? Maybe our specific salesman *forgot* to mention a certain weakness that pertains to us? NOTICE: This comment was moved from an existing discussion: Evolv Detects Certain Chromebooks As Weapons Evolv Detects Certain Chromebooks As Weapons By Nikita Ermolaev and Donald Maye, Published Jan 03, 2022, 09:43am EST | Email This Evolv detects certain Chromebooks as weapons, one of a series of weaknesses that the publicly traded company has advocated IPVM not to report on. IPVM Image Inside this note, based on an Evolv's salesperson presentation to a US school district, we examine what this weakness is, why Evolv struggles to distinguish certain Chromebooks from actual weapons, and how Evolv aims to sell around this issue. We also include a response from Evolv's management. This is the first in a new series examining Evolv's design, weaknesses, advertising, and risk to public safety. Chromebook False Alerts Disclosed To School District In a presentation to an Illinois school board, an Evolv salesperson and school administrators discussed the issue of Chromebook false alerts: The salesperson also proposed several workarounds, such as students passing the Chromebooks around the system or holding them above their heads. In a response to IPVM, Evolv stated that such workarounds allow the false alerts to be isolated while maintaining the flow through the screening area. Many school systems distribute laptops to their students. Depending on the construction, some laptops alert in the system and others do not. For all customer installations, we provide suggestions to incorporate the technology into their security processes and operations. For example, some schools ask their students to hold their laptop as they walk through the system. The alert can quickly isolate the laptop to minimize the friction around screening, while also detecting any potential weapons. Maintaining flow through the system is important to the safety and security of students and educators, as large queues that can become soft targets. Why Evolv Labels Chromebooks As Weapons While Chromebooks do not look like guns or knives, at least to humans, to Evolv's system they do, because of the hinges inside certain models. The school board and district's CIO, who tested the system, discussed with Evolv the limitations associated with Chromebooks, with the CIO stating: Yeah, it's the hinges on the Chromebooks because of the tubularsteel, look kind of like a barrel. The image below highlights the internal hinges of a Chromebook. IPVM Image Chromebooks are used widely by students in schools. The Covid-19 pandemic necessitated remote learning, which led many US schools, e.g., to utilize Cares Act funding to equip students with Chromebooks. Given many schools have a high volume of Chromebooks passing through their doors, false alert rates can present a significant objection, with the district's CIO noting a high false alert rate observed during testing. Evolv Has Not Solved This False Alert Given the recommended workarounds to bypass false alerts of Chromebooks and Evolv's statement to IPVM, it is clear there are accuracy limitations in Evlov's system that are not solved. Previously, Evolv blamed their system being "subject to the unyielding laws of physics" and that it was "impossible to fix every vulnerability". How Evolv Sells Around This Issue While the Evolv salesperson discussed limitations associated with Chromebooks, he sold around the issue in three ways: Introducing workarounds for end-users - passing Chromebooks around the system or raising them above a student's head Suggesting the system could eventually be trained to miss Chromebooks Sharing how certain Evolv Express sensitivity settings should not alarm on Chromebooks The Evolv salesperson elaborated on changing sensitivity settings, stating that certain end-users adjust the settings at different locations or times of day and that one setting will miss Chromebooks entirely: And you know, they run it on that at certain doors. They don't run it on those certain setting every day they switch it up. It's kind of they kind of play it how they want to play it. But there is a setting if you wanted to put it on that it'll miss most, it should miss most Chromebooks. However, changing sensitivity settings presents trade-offs for end users, as the system will become less accurate in detecting certain threats. In upcoming reports, IPVM will examine the tradeoffs of using different Evolv sensitivity settings, their impact on false alerts, and what weapons may be missed.