a notable point in here, particularly given the recent WCK murders:
In an unprecedented move, according to two of the sources, the army also decided during the first weeks of the war that, for every junior Hamas operative that Lavender marked, it was permissible to kill up to 15 or 20 civilians; in the past, the military did not authorize any “collateral damage” during assassinations of low-ranking militants. The sources added that, in the event that the target was a senior Hamas official with the rank of battalion or brigade commander, the army on several occasions authorized the killing of more than 100 civilians in the assassination of a single commander.
When you abstract out pieces of the puzzle, it’s easier to ignore whether all parts of the puzzle are working because you’ve eliminated the necessary interchange of information between parties involved in the process. This is a problem that we frequently run into in the medical field and even in a highly collaborative field like medicine we still screw it up all the time.
In the previous process, intelligence officers were involved in multiple steps here to validate whether someone was a target, validate information about the target, and so on. When you let a machine do it, and shift the burden from these intelligence officers to someone without the same skill set who’s only task is to review information given to them by a source which they are told is competent and their role is to click yes/no, you lose the connection between this step and the next.
The same could be said, for example, about someone who has the technical proficiency to create new records, new sheets, new displays, etc. in an electronic health record. A particular doctor might come and request a new page to make their workflow easier. Without appropriate governance in place and people who’s job is to observe the entire process, you can end up with issues where every doctor creates their own custom sheet, and now all of their patient information is siloed to each doctors workflow. Downstream processes such as the patient coming back to the same healthcare system, or the patient going to get a prescription, or the patient being sent to imaging or pathology or labs could then be compromised by this short-sighted approach.
For fields like the military which perhaps are not used to this kind of collaborative work, I can see how segmenting a workflow into individual units to increase the speed or efficiency of each step could seem like a way to make things much better, because there is no focus on the quality of what is output. This kind of misstep is extremely common in the application of AI because it often is put in where there are bottlenecks. As stated in the article-
the goal here is purely to optimize for capacity, how many targets you can generate per day, rather than on a combination of both quality and capacity. You want a lot of targets? I can just spit out the name of every resident in your country in a very short period of time. The quality in this case (how likely they are to be a member of hamas) will unfortunately be very low.
The reason it’s so fucked up is that a lot of it is abstracted yet another level away from the decision makers. Ultimately it is the AI that’s making the decision, they are merely signing off on it. And they weren’t involved in signing off on the AI, so why should they question it? It’s a dangerous road - one where it becomes increasingly easy to allow mistakes to happen, except in this case the mistake can be counted as innocent lives that you killed.