What I've gained, and lost, from chatting with my documents
Knowledge begets more knowledge with PDF query tools
AskYourPdf is the ChatGPT plugin that promises so much. The clue is in the name: feed the chatbot any document and, aided by the plugin, it will duly answer any questions you may have. In the past I have flirted with other corner-cutting reading tools like GetAbstract, which provided 5-page, neatly bulleted summaries of books, compiled by a human reader. My use of those earlier tools was fleeting because this ultra-processed approach to reading leaves out so much - most of all, the opportunity for me to dig into these texts, connect with the writer’s communicative intent and extract my own insights.
The value proposition of AskYourPdf runs deeper than its antecedents, however, because it does not merely present a static summary. Instead it turns reading into a conversational dialogue, the length and scope of which is bounded only by one’s prompts.
From my first encounters with these tools, it is impossible to deny their potential in enriching my understanding of a text. But, as is typical of generative AI, the promised efficiency gains are not without their trade-offs.
Your chat needs a truth-checking mechanism
The use cases of AskYourPdf are varied. I first aimed the tool at a technical paper, wondering if it could clarify some of the thornier details. The paper in question claims to have developed a model that achieved a success rate of 78% success rate on a set of maths problems. The precise details of how this metric is defined eluded me on first reading. But after some prompting to AskyourPdf, the veil was lifted and I felt wiser for it.
My moment of enlightenment did, however, prompt the unnerving question of how I could be sure of the veracity of AskYourPdf’s response. What if this was just another of its concoctions, an explanation both plausible and incorrect?
The only way out of the confabulatory rabbit hole of large language models is to have some mechanism for truth-checking. A reader must come equipped with domain knowledge; they must grasp the underlying concepts to such an extent that they can evaluate a chatbot’s outputs. One cannot go into a paper blind and expect to come out the other side the sage.
One (no doubt temporary) limitation of AskYourPdf is that it does not provide references to the source material. Alternatives like ChatPDF cite page numbers within their responses. Implicit here is the need for the reader to return to the source material and engage with its finer details. As I did so, new questions were triggered in my mind, leading to a virtuous back-and-forth between directing queries at the chatbot and diligently working through the paper.
This does bring into question the stated promises of these tools. AskYourPdf claims to be a ‘time saver’ yet my time was not saved as much as it was redistributed - with my understanding of the paper enhanced, it was surely time well spent.
The hard graft of reading
When we approach PDF query tools with purpose and a prior grasp of the material, they show their worth. I have them at the ready, for instance, when grappling with the rules of a new board game. I am a sucker for high-complexity rules, but it inevitably results in errors on my part. Where previously I would rely on prolonged YouTube tutorials and forums to fill my knowledge gaps, now I can fire away my query at ChatPDF and plug core gaps in my knowledge.
The tool is not a substitute for the rulebook; there is still the honest labour of thumbing through the pages and absorbing the rules at surface level, so that I know what clarifications are worth seeking and whether the proposed answers are legitimate. There will always be a role, too, for human-made YouTube tutorials that animate my understanding of the rules. But with ChatPDF by my side, my error rate has decreased markedly, much to the relief of my fellow players.
A more knowledgeable reader makes for a more powerful plugin
Like much of generative AI, the value of PDF query tools is proportional to the amount of knowledge and expertise one brings to their interaction. One text I hope I command an expert grasp of is my own book, Mathematical Intelligence. When I fed the manuscript to ChatPDF, and asked it leading questions such as ‘what single aspect of mathematical intelligence does the author claim is least automatable?’ it takes the bait, offering a plausible if slightly vague response based on a particular passage in one of the early chapters. A cursory read of the book will tell you that the question is ill-defined, because I make no attempt in the book to pit traits against one another, or to single any of them out in these terms. It is a mean-spirited prompt to be sure, but a useful reminder that these tools have no meaningful relationship to “the truth”, and no external ground for evaluating the meaning of our words; hence their propensity for “bullshittery”.
When I switched tack and asked ChatPDF to list three points that challenge the main arguments of the book (which I did not even specify in the prompt), the outcome was altogether more interesting. The points raised have indeed cropped up in podcast exchanges, leaving me with plenty to reflect on. The collaborative potential of these tools - in this case, as a contrarian sparring partner - is evident enough, and bolstered by the prior knowledge I already bring to the interaction.
In sum…
Efficiency is not the metric to measure these tools against - there is no 10x gain in knowledge on offer here. These tools are lost on text that you have not already examined in some detail; you will be left at the mercy of confabulation, ill-equipped to ask the right questions or probe outputs. If it is quick-fire summaries you are seeking, remember that they usually exist already in the form of abstracts and synopses. For anything deeper, there is no alternative but to get stuck into the text.
It is then, and only then, that you will realise the benefit of PDF query tools. Another of AskYourPdf’s claims is that it makes learning ‘playful’, and that it will ‘keep you coming back for more’ - that much is true, provided you are willing to embrace your role as a reader and not simply delegate it to a chatbot.