General-purpose AI vs legal AI: four things that break when an Indian advocate uses ChatGPT for court work

Most writing about AI and Indian legal practice fixates on one failure — the invented citation — and then reassures you that careful checking solves it. Fabricated authority is real, and Indian courts have now ruled on it. But it is the most visible problem, not the most dangerous one, precisely because it is visible.

Three quieter failures matter more, and one of them is routinely described wrongly in exactly the articles advocates are most likely to read. This piece sets out all four, with the provisions, and says plainly which constraints actually bind an Indian advocate today and which do not.

1. Citations: the failure everyone knows about

A language model produces a citation the same way it produces any other sequence of words. It has no index of judgments to consult and no mechanism for reporting that nothing matched. Asked for authority on a proposition, it generates what authority for that proposition would plausibly look like.

That is why the failures do not look like errors. They look like Ayinde's fake Court of Appeal decision, or the four fictitious "Supreme Court" precedents that reached a Vijayawada trial court order, or the twenty-one fabricated quotations out of twenty-three in a Californian appellate brief. Correctly formatted, confidently stated, entirely invented.

On 2 July 2026 the Supreme Court of India held in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd., 2026 INSC 668:

"It is a misconduct on the part of an advocate to cite such judgments without verification."

The Court also held that a decision touched by such material is set aside "even if an iota of fake or hallucinated material enters the decision-making process." We covered the full Indian record — every documented matter, and the uncomfortable fact that in five of nine the fabrications came from the bench rather than the bar — in a separate article.

The architectural point is what matters here. A system that retrieves from an indexed corpus and then writes from what it retrieved cannot invent an authority, because there is nothing to invent from. A system that generates citations from a model's parameters can do nothing else.

2. Privilege: the failure nobody discusses

This is the one that should worry you more, because unlike a fake citation it leaves no trace in the filing.

What the privilege actually says now

The Indian Evidence Act 1872 was repealed on 1 July 2024. Section 126 — the professional-communications privilege — is now Section 132 of the Bharatiya Sakshya Adhiniyam, 2023:

"No advocate, shall at any time be permitted, unless with his client's express consent, to disclose any communication made to him in the course and for the purpose of his service as such advocate, by or on behalf of his client, or to state the contents or condition of any document with which he has become acquainted in the course and for the purpose of his professional service..."

The obligation survives the retainer: the Explanation provides that it "continues after the professional service has ceased."

On the Bar Council side, two rules under Chapter II of the Bar Council of India Rules bear directly:

Rule 17. An advocate shall not, directly or indirectly, commit a breach of the obligations imposed by Section 126 of the Indian Evidence Act.

Rule 24. An advocate shall not do anything whereby he abuses or takes advantage of the confidence reposed in him by his client.

Rule 17 as published still names the repealed Act — the reference falls to be read as the re-enacted provision, though it is worth checking whether the BCI has since amended the text. Breach of these rules is the route by which a confidentiality failure becomes a matter under Section 35 of the Advocates Act 1961, where the disciplinary committee may reprimand, suspend, or remove an advocate from the State roll.

The gap that catches people

Here is the provision that decides the AI question, and it is almost never quoted. Section 132(3) BSA:

"The provisions of this section shall apply to interpreters, and the clerks or employees of advocates."

That is the whole extension. It modernised the old wording — the Evidence Act said "servants" — and it did not widen the class.

A cloud AI vendor is not your employee. Not an interpreter, not a clerk. The privilege that covers your junior does not travel to a third-party service you send the brief to. There is no Indian authority construing this in the AI context, because the question has not yet been litigated. But nothing in the text suggests independent contractors, SaaS providers or model APIs fall inside it, and the 2023 re-enactment was the moment to widen it if Parliament had wanted to.

What the vendors actually do with what you type

This is verifiable, and the answers differ sharply by tier.

Two things in that table deserve to be pulled out.

The "just use the API" advice fails for Google. Google's API terms state that for unpaid services, including Google AI Studio and the unpaid Gemini API quota, it uses submitted content to improve its products, that "human reviewers may read, annotate, and process your API input and output", and — in Google's own words — "Do not submit sensitive, confidential, or personal information to the Unpaid Services." Billing status, not payment, determines the tier: an advocate experimenting in AI Studio on a bare Google account is on the training side of the line. Google grants paid-tier treatment on free services to users in the EEA, Switzerland and the UK. India is not on that list.

Google's human-review retention survives deletion. The Gemini Apps privacy notice states that chats reviewed by human reviewers "are not deleted when you delete your activity" and are retained for up to three years. The same notice carries an instruction that reads oddly next to a privileged brief:

"Please don't enter confidential information that you wouldn't want a reviewer to see or Google to use to improve our services, including machine-learning technologies."

And a caveat that applies to every vendor and every tier: the thumbs-up / thumbs-down feedback button is a training opt-in. On Anthropic it captures the entire conversation for five years, on commercial plans included. Tell your staff not to press it.

Where the data physically sits

As of September 2026, Vertex AI in the Mumbai region is the only configuration among these three that keeps both storage and inference inside India, and only for a short list of models. OpenAI's India residency covers data at rest; the prompt still leaves the country to be processed. The global endpoint is the default in most SDKs and offers no residency guarantee at all.

The one Indian instrument that addresses this directly

The Kerala High Court's Policy Regarding Use of Artificial Intelligence Tools in District Judiciary, dated 19 July 2025, is the only binding Indian instrument on cloud AI and confidentiality. By its own clause 2.1 it applies to the district judiciary and their staff — it does not bind advocates. But clause 4.2 is worth reading anyway:

"Most of the AI tools, including the currently popular GenAI tools such as ChatGPT and Deepseek, are cloud-based technologies wherein any information input given by the users may be accessed or used by the service providers concerned to advance their interests, including fine-tuning of their models. Submitting information such as facts of the case, personal identifiers, or privileged communications or uploading any other documents relating to the litigations to any such AI tools may result in serious violations of confidentiality. Hence, the use of all cloud-based services should be avoided, except for the approved AI tools."

That is a High Court's own analysis of the risk. It governs judges. An advocate — whose duty of confidence to the client is, if anything, stricter than a judge's — would find it awkward to explain why a practice the Kerala High Court forbids its own judiciary is unobjectionable in chambers.

3. The data-protection claim that is simply wrong

You will read, in a great deal of Indian legal-tech marketing, that putting client data into a consumer AI tool exposes a firm to penalties of up to ₹250 crore under the Digital Personal Data Protection Act 2023.

That is not the law today. The correction matters, because a reader who acts on the false version and later discovers it will discount everything else they were told.

The DPDP Rules 2025 were notified in November 2025 with staggered commencement written into Rule 1. The tranches:

So today: no notice obligation, no consent obligation, no statutory security- safeguards duty, no breach-notification duty, no erasure duty, no grievance- redressal duty, and no penalty Schedule in force. Section 33, which the Schedule hangs off, is in the May 2027 tranche.

There is a further, sharper fact. The Data Protection Board of India was established by notification in November 2025 — and as of 1 September 2026 it has no appointed Chairperson and no appointed Members. The Government invited applications in May 2026. The enforcement organ of the Act exists, as one analysis put it, "only as a statutory possibility."

Assent (Aug 2023)0

Rules notified (Nov 2025)27

Consent Manager rules (Nov 2026)39

Obligations + penalties (May 2027)45

Digital Personal Data Protection Act 2023 (No. 22 of 2023), assented 11 August 2023. Commencement per the DPDP Rules 2025, Rule 1(2)-(4), which tie the 12- and 18-month tranches to the date of gazette publication in November 2025. The Data Protection Board had no appointed members as at 1 September 2026.

What will apply, when it does — and the part that survives the exemption

When the substantive provisions commence, most litigation work is exempt from the consent machinery. Section 17(1)(a) disapplies Chapter II and Chapter III where

"the processing of personal data is necessary for enforcing any legal right or claim".

That switches off notice, consent, erasure, grievance redressal and the cross-border restriction for the processing you do to run a case.

But read the opening words of Section 17(1) carefully:

"The provisions of Chapter II, except sub-sections (1) and (5) of section 8, and those of Chapter III and section 16 shall not apply where—"

Two duties are carved back out of the carve-out. Section 8(1) — you remain responsible for processing done on your behalf by a processor. And Section 8(5) — you must take reasonable security safeguards to prevent a personal data breach.

Section 8(5) is entry 1 in the Schedule. It is the provision carrying the ₹250 crore maximum. So the accurate statement, for May 2027 onwards, is the opposite of the one usually made: litigation work is largely exempt from DPDP's consent regime, and never exempt from the duty to secure the file — which is the most expensive duty in the Act.

Two further limits worth knowing. The exemption is purpose-scoped: it covers processing "necessary for enforcing any legal right or claim", so marketing lists, CRM records, prospective-client enquiries and HR data sit outside it and attract the full regime. And there is no authority construing its boundaries, because the section is not yet in force.

4. The code transition, and what a general model does with it

The Bharatiya Nyaya Sanhita, the Bharatiya Nagarik Suraksha Sanhita and the Bharatiya Sakshya Adhiniyam came into force on 1 July 2024, replacing the Indian Penal Code, the Code of Criminal Procedure and the Evidence Act. Which code governs a given matter depends on when the offence occurred, so for years yet every criminal practice runs both systems in parallel.

This is a hard problem for a general-purpose model for a specific reason: its training data is overwhelmingly the old codes. Nearly two centuries of Indian criminal jurisprudence, textbooks, commentaries and judgments are written in IPC and CrPC numbering. The new numbering has existed for two years. A model predicting the most probable next token, asked for the provision on a given offence, is heavily weighted towards the answer that was right until July 2024.

The incumbent legal databases have the same problem in a different form. One major Indian research platform carries over three thousand doctrinal essays of which none mention the BNS, while a hundred and forty-five are titled to IPC sections and a hundred and twenty-eight to CrPC — and it stopped publishing essays in mid-2025. The corpus is not wrong so much as frozen before the transition.

What actually works is boring and mechanical: an index that holds both numbering systems and the mapping between them, so a search on the section you know returns the section that now governs. That is a database problem, not a reasoning problem, and it is not one a language model solves by being larger.

What this means in practice

Stripped to the operative points:

• The binding constraints on an Indian advocate today are professional, not regulatory. BSA Section 132, BCI Rules 17 and 24, Section 35 of the Advocates Act, and — since July 2026 — the Supreme Court's holding that citing unverified AI output is misconduct. Not the DPDP Act, which does not bite until May 2027.
• Privilege does not follow the file to a vendor. Section 132(3) reaches interpreters, clerks and employees. It does not reach a SaaS provider.
• Consumer tiers train on your input by default at all three major vendors. Business and API tiers largely do not — except Google's unpaid tier, which does.
• There is no Bar Council guidance on any of this. The Supreme Court directed the BCI to write some in July 2026. Until it exists, you are applying a 1961 Act and rules drafted for paper briefs to a problem neither contemplated.

What a legal-specific system changes

Not everything. It does not remove your duty to read what you file, and any vendor claiming otherwise is selling you the risk rather than removing it.

What it changes is which failures are possible.

Citations are retrieved, not generated. Synapse fetches from an indexed corpus of Indian statute and judgments before the sentence is written, and the draft is assembled from what came back. Nothing is cited that was not retrieved. Every line opens to its source — the provision, the paragraph of the judgment, or the moment in the consultation where your client said it.

Both numbering systems are indexed. Search by the section you know and get the provision that now governs, in either direction.

The confidentiality question has an actual answer. Client data is held on India-hosted infrastructure. Personal identifiers — names, Aadhaar numbers, phone numbers, case numbers — are tokenised before any model call leaves that boundary and rehydrated on the response, so what reaches the model is [PARTY_1] and not your client. That is a design decision about where the data-residency line sits, and it exists because the answer to "is the vendor covered by privilege?" is no.

The question worth asking of any legal AI tool is not how good its output looks. It is what happens when the answer does not exist — and what leaves your chambers when you ask.

---

Statutory positions stated as at 1 September 2026: DPDP Act 2023 (No. 22 of 2023) and DPDP Rules 2025; Bharatiya Sakshya Adhiniyam 2023, in force 1 July 2024; Bar Council of India Rules, Chapter II; Advocates Act 1961. Vendor terms were checked on 1 September 2026 and change frequently — verify before relying on any row in the tables above. This article is general information about legal technology and practice, not legal advice, and does not create an advocate–client relationship.