Doc automation: Is smarter always better?
Why I think traditional doc automation platforms are superior to Gen-AI
Gen-AI is rapidly changing the world of software, every day more and more platforms announce the addition of gen-AI tooling whilst more and more gen-AI companies pop up.
This is a fairly normal pattern for new technology hype cycles - smartphones and blockchain saw a similar reaction from companies trying to work out how these things could augment their offering and other companies offering XX-but-on-mobile (as is hopefully clear by now, not every company needs their own mobile app - we have limited screen real estate and I’m not using that to install mobile apps for every shop or venue I visit or interact with - if you’re a pub that requires me to install an app to order at my table, we can’t be friends).
Gen-AI is a slightly different hype-cycle, as it genuinely does unlock capabilities that just didn’t exist before. For any platform or system that deals with written words (knowledge, user interaction etc) gen-AI does open up new features and savings that can be huge. So it comes as no surprise that gen-AI is being thrown at lots of document tooling platforms, drafting, reviewing, analysing - gen-AI can be applied to it all.
So should we be using gen-AI for document automation? (Spoiler alert: No, we shouldn’t)
An intro to document automation
Doc automation is a concept that has been around for ages (I remember using the mail-merge functionality in MS Word way back in the 90s, and thats basically the same idea).
The concept itself is simple: You have a template of some document and you insert placeholders for names or sections that should be customisable - for example, if you had a template for an employment contract you might have placeholders for employee name, salary, job title, address etc. From there, you have a system whereby you just provide the values for those placeholders and mashes them together to generate a completed document. In our employment contract example, every time you have a new employee you put in their details and out comes the contract.
The benefits of an automated employment contract are probably apparent:
You don’t need to go back to your legal team every time you need a new contract
Its super quick (instantaneous, effectively)
You don’t need to worry about errors or inconsistencies (once you are happy with the initial template)
If you need to make contract updates, update the template, re-generate them all and your are good to go
Why isn’t everyone using doc-auto?
With our employment contract example, it sounds like a no-brainer. You’d think everyone in any industry where they have to create repeatable documents (law, for example) it’s an obvious solution.
However, for documents approaching even a vague amount of complexity, it actually becomes quite hard to build systems that can both 1) handle the complexity 2) be manageable and have a positive user-experience.
Our employment contract example requires a basic template as a starting point, and then we add in a couple of placeholders and we might be most the way there. Technically, writing software that can replace placeholders in a template is super simple.
But now let’s say we want to add a specific clause about home-working for any employee that is going to be working remotely. Next, maybe we need to change some wording of that new clause if it is hybrid working. On top of that, we want it to reference another newly added clause about hybrid working (all with correctly updated and specific clause numbering, of course).
You can see it starts to get complicated.
But it doesn’t stop there though, these conditions might be nested - that is, we might have a question that determines if we include an entire section to the document, which might then have more additional clauses/sections that are driven by other questions (and this can keep going - there is no fixed limit on how many nested conditions we might end up with).
Next up we need to start dealing with contracts that involve multiple parties, and we have to deal with repeating sections - oh, and those repeating sections need to have party-specific conditional clauses or behaviours?
Gen-AI, to the rescue!
Dealing with documents and large volumes of text is gen-AI’s super power, so of course, people might be tempted to look to gen-AI to solve doc-auto’s challenges.
We can imagine a couple of different potential gen-AI solutions to doc automation:
Draft generation + clause recommendations: An in editor solution that either generates an entire first draft or one that suggests clauses you should add as you type/structure the document (Microsoft’s Word Copilot plugin can offer similar capabilities).
Conversational doc-auto: You upload an existing contract or template to a gen-AI LLM and it identifies the relevant values that should be changed (names etc) and in a ChatGPT style interface asks you what values should be inserted, and at the end spits out a completed contract.
Burden of doubt, strikes again
So should we all just use gen-AI and forget about the complexity of complex documents and let the robots do the work? No, I don’t think so, at least.
Once again, our old friend is back, the achilles heel of generative AI, the burden of doubt (I’ve written about this concept a few times).
Generative AI is non-deterministic, which is generally not a desirable feature in expert knowledge domains where accuracy is paramount:
If you attempt to generate the same contract, with the same input, twice over, you might get different documents.
Once it’s generated, because its non-deterministic, someone has to essentially check every line of the generated document for accuracy - this is the burden of doubt cost - because there is a chance that it might have done something random, we have to check it all.
Of course, going from a blank page to a v1 in a matter of minutes is a nice saving in itself, so compared to a human drafting that first version, it’s an improvement, but that is where those savings end.
Let’s compare this to a deterministic, traditional doc-auto solution:
If you enter the same values (names, parties etc) - you are guaranteed to get the same output every time
Assuming the template is correct, you only need to check the input values, you do not need to review the entire document, line-by-line.
In this scenario, traditional doc-auto brings you that blank-page-to-v1 savings but also huge savings on review and amendments.
As an analogy to illustrate the point - imagine you have an automated car assembly line. We have multiple machines/robotic arms that are setup to repeatedly do their one job (you might consider them “dumb” in this sense) - the assembly line is effective, deterministic, reliable and has a high throughput.
Would you ever want to have any part of that line non-deterministic? Of course not! If any single part of this assembly line has variation in what it might do, or might skip, or alter, it is going to cause a lot more overhead for the factory.
In an interview with Artificial Lawyer, Linklaters’ head of legal tech, Shilpa Bhandarkar was quoted talking about the use of generative AI in such domains (emphasis is mine):
GenAI, when it works, is amazing. It can do in seconds what otherwise took hours. It also really helps with what Marc has coined ‘blank-page-itis’ – we seem to find editing a first draft much easier than having to draft from scratch, and GenAI has been very helpful in giving us a first draft to work from.
The big proviso though, is the ‘when it works’. It does hallucinate, it can be annoyingly random in its responses, and it is very difficult to verify sources.
Traditional doc automation solves the “blank-page-itis” problem but without the non-deterministic hallucination problem.
Doc automation doesn’t need rescuing
But I thought you said doc auto was too hard?
I did, but really those were just a list of the perceived challenges and challenges that folks have encountered over the years experimenting with doc auto.
Thankfully, it is 2024 and there are now great doc auto platforms - platforms that have built great solutions to the complexity and user-experience challenges.
For my money, if you can get gold-standard document/contract templates, and a system that can repeatedly and reliably generate new versions, then that’s the far better bet. Even if you are just using it to get from blank page to v1, and solving the above “blank-page-itis” problem, it’s the cheaper and more effective solution.
But even if we did imagine a future scenario where people had managed to build a gen-AI doc-automation system that all but ruled out the hallucinations and non-deterministic behaviours (or maybe more realistically got it working 99.999% of the time) - it still doesn’t offer meaningful advantages, and is basically just operating at the same level of traditional doc automation, so the question then is why are we even bothering?
Ok, so we should not bother with gen-AI doc tools?
It is far more useful to think about processes and work-flows from a holistic, top-down point of view. Doc automation fits very specifically into a part of the document workflow, and as should be clear, is the best tool for that job - but there are plenty of other parts of the end-to-end workflow where generative AI tooling can further improve the process.
We do not need an either-or solution, we want to utilise tools that streamline and improve the efficiency of the entire process - for repeatable document drafting, traditional doc auto is my bet, but there are other areas where the addition of generative AI tools might be useful:
Reviewing existing or inbound contracts
Interrogating or asking questions of a document, extracting information
Adding specific, custom clauses or wording to a standard generated contract
Highlighting potentially problematic or important wording in a contract
Some of these features are highlighted in one of the latest OpenAI promotional videos, where Moderna talk about their usage of an OpenAI GPT called Contract Companion for analysing and summarising contracts:
Conclusion
To paraphrase an old saying when you have a fancy new hammer, everything looks like a nail - There are lots of applications for generative AI to fill gaps and unlock previously unavailable functionality, but we should take care not to overlook traditional technologies and solutions where they are still superior.
