This is despite findings that many business leaders in the UK (81 per cent) and the USA (89 per cent) worry about getting software projects done on time.
Recent studies for the new book "Impact Engineering" reveal that 65 per cent of software projects that use Agile methods to plan and manage requirements don't meet deadlines or budgets, and they don't hit quality targets either.
On the other hand, projects that use the Impact Engineering method, which is explained in the book released today, only fail 10 per cent of the time.
The study's data is so convincing that the chance of it being wrong is like rolling a six on a dice six times in a row on your first try.
Dr. Junade Ali and J.L. Partners did the research with 600 software engineers (250 from the UK and 350 from the USA) between 3-7 May 2024. J.L. Partners is part of the British Polling Council and sticks to its rules.
The Agile Manifesto values things like "Working software over comprehensive documentation," "Customer collaboration over contract negotiation," and "Responding to change over following a plan."
But the new study found that projects with precise specs or requirements set before starting were 50 per cent more likely to succeed. Projects with well-defined requirements from the get-go were 97 per cent more likely to succeed, and those that didn't have to change requirements too much late in the game were 7 per cent more likely to succeed.
Other good practices helped, too. Projects, where software engineers felt safe talking about and fixing problems right away, were 87 per cent more likely to succeed, and projects based on real-world issues were 54 per cent more likely to succeed.
Interestingly, the study didn't find a big difference in success rates between engineers working on one project or several, even though cutting down on ongoing work is a big part of the Lean software development approach. However, Dr. Ali's earlier research showed that 83 per cent of software engineers feel burned out, mainly because of too much work.
This research comes as people are more aware of big software failures. Dr. Ali's previous book, "How to Protect Yourself from Killer Computers," looked into deadly software issues caused by design flaws, such as plane crashes, car accidents, and hospital radiation overdoses.
For example, the Horizon IT system was one of the first big projects to use Agile methods, specifically Rapid Application Development. But Fujitsu engineers in a public inquiry said this approach caused technical issues because it lacked a solid process for setting requirements.
Charles Cipione, a technical expert at the inquiry, said, "If you don't have a good design, it won't work right." Not fixing these problems and trying to hide them led to the Post Office scandal, the worst miscarriage of justice in British history, with many people, including a pregnant woman, wrongly jailed.
The study also found that UK software engineers are 13 per cent less likely than their US counterparts to feel they can talk about and fix problems. This is the biggest difference in practices between the two countries. It follows a 2023 study by Engprax that found 75 per cent of UK software engineers faced backlash the last time they reported something wrong.
Dr. Junade Ali, author of "Impact Engineering," said: "Since 65 per cent of Agile projects aren't delivered on time, we need to rethink Agile's popularity. Our research shows that delivering top-notch software on time and within budget depends on a solid process for setting requirements and being able to talk about and solve problems as they come up, while also preventing developer burnout. This is what Impact Engineering is all about."
"Impact Engineering" is now available on Amazon as a Kindle eBook and paperback. The book uses real-life stories of people and companies changing for the better through Impact Engineering. It also introduces a new psychological approach to successful change and explains the science behind the method.