Let me start out by saying that this is not any kind of walk-back on my previous claims of doc-automation superiority. If I was presented with two systems for generating documents, all things being equal, but one of them was deterministic and the other was non-deterministic and 99.999% reliable then I’m picking the former, every time, because why wouldn’t you? If you disagree with that starting point then I really don’t know what to tell you.
Since my article last week (Is smarter always better? Doc automation vs Generative AI) there have been some good points raised in follow up discussion (as well as some brazen mocking of my position in the face of fairly impressive gen-AI capabilities 😂), and I thought it’d be fun to get into that a bit more.
Why aren’t more people using doc-automation?
One point that came up a couple of times on LinkedIn discussion following my last article was that I had skipped over one of the key reasons people aren’t using doc-automation. That is, despite the doc-auto platforms being far better now, it still takes considerable resource investment and expertise for an organisation to implement a doc-automation project.
This is an excellent point, and one that I didn’t really get into in that last newsletter.
The proposed value in doc-automation is its ability to scale - you pay an upfront cost of setup/purchase and then whether you generate one or a million documents, its reliable, deterministic and ongoing cost per document remains low (effectively zero).
If we look at this, it might be fair to think that generative AI could disrupt this model by taking away the upfront investment required to get up and running. However, if we use a generative AI solution, whilst the upfront investment requirement is removed, there is a linear, ongoing time & effort cost of double checking the generated output (sure, this might be a small cost and probably diminishes over time as confidence increases in the product’s success rate)1. The tradeoff becomes upfront cost or ongoing cost.
At the moment, it might be that that upfront setup cost of traditional doc-automation is so high that its never realistically undercuts the cost of double checking non deterministic output. Alternatively it might be that the effort to double check and review AI generated documents is so negligible that it takes years before it becomes a meaningful cost.
But those specifics aside, one thing we can say for certain is that if we can get that upfront cost of traditional doc-auto setup down to a negligible amoount, then in almost all cases, its the better option.
Across domains, people are out trying to find products and problems that generative AI can solve. But the key point is this: the problem that needs solving is not doc-automation, it’s the cost of setup.
A brief detour into product management
I say a detour, if it wasn’t already obvious, this entire thing is a vaguely veiled rant about product management and how to build, and think about, products.
The above quote is often thrown around in product management talks and articles, and I think it applies here. People don’t want better doc-automation, they want easier/lower-cost setup. The doc-automation part is good. So let’s not get caught up trying to use generative AI to make a better doc-automation, and instead use it to solve the actual problem: the upfront costs.
If Carling did doc-automation
My response on LinkedIn to the previously mentioned comments was the same every time:
If I were building a doc automation platform (which Im not!), my main focus right now would be:
1. Gen AI tooling to improve the template + coding process - whether that be a copilot type experience or something that could take entire docs (or entire data sets) and template them (would probably need reasonable guard rails and probably a multi step process in terms of LLMs and checks)
2. Looking at the wider end-to-end process and what else I could build to compliment the doc-auto process (the more gen AI focussed tools such as review/summarisation/interrogation/etc) - now there is so much interest in those kind of tools, being able to offer a joined-up experience across the entire document workflow process is only going to make it more compelling
Rather than trying to replace doc-automation with a less reliable (albeit eventually very marginally), non-deterministic, generative AI process - I think there is an opportunity right now to combine generative AI with traditional doc-automation technologies to build a transformative product in the document space.
We know gen AI can write code, we know gen AI is good at drafting documents - why not use it to solve the expensive setup costs of traditional doc-automation? Build tools that enable organisations to quickly convert their document suites into automated templates. That one-off setup process can benefit from gen AI efficiencies and is well suited to human-in-the-loop curation and oversight, as it’s a limited time operation that doesn’t need to scale.
The second point I covered in my previous article, but it’s an important second part of the puzzle. Use gen AI to improve the setup process, and then pair the technologies together for other steps in the end-to-end document workflow process. For example, you could have your doc-automation tool output the document in a smart editor that lets you have further gen AI tooling customise it, add specific clauses (with redline), compare clauses or check guidance on the drafting. Why wouldn’t you want a document software that lets you review, summarise, interrogate documents as well as reliably generate entire documents or suites of docs?
So do I actually think Doc-Automation wins out over the next 5+ years?
No.
Superior quality and reliability are not the markers by which this battle is being fought. Partly because of the usual market forces (FOMO and market hype), but also because generative AI tools will be bundling lots of cool tools and efficiencies together and will start to make it much harder for a dedicated doc-automation vertical to exist in its own right. Once you have a generative AI tool that can do lots of things including documents (albeit needing checking), then the perceived savings of doc-automation is reduced a lot.
I think it’s feasible to imagine a superior product that would include traditional doc-automation alongside generative AI capabilities (see above), but we will have to wait and see if this comes about.
Caveat: Off-the shelf doc-automation
Off-the-shelf doc-automation, e.g. where you simply pay a platform for already automated documents obviously entirely side steps the initial set up cost/resource investment requirement of doing it yourself, whilst also having the benefit of low-burden-of-doubt, deterministic results.
A brief detour into product management part 2
A pet peeve of mine, and probably lots of people, is when an underlying technology is pitched as a product.
Document automation is the product - whether it has mind-bending AI under the hood or it’s just a bunch of humans behind the scenes a la The Wizard of Oz doesn’t (and shouldn’t) matter to the customers using it.
Technical implementation details aren’t a product - the outcome is the product. How that particular platform implements the outcome is by-the-by, what matters is price, ease of use, efficiency, functionality and reliability.
Don’t buy a product because it’s built on AI. Buy it because it solves a problem you need solving at a price that is worth you paying.
For the more technically minded of you, to describe this in Big-O notation, traditional doc automation costs O(1) and generative AI document automation costs O(n)