Most AI assistants follow a route that has by now become almost invisible to us. You type a question into your browser, the question travels across the internet to a server, a large model somewhere in a data centre processes it, and an answer makes the journey back to your screen. It is an impressive arrangement. It is also a little like sending every arithmetic problem to a very efficient, and presumably rather expensive, library.

Chintan started with a slightly different question: could a useful AI assistant run mostly on the visitor’s own computer?

The short answer is yes. The slightly more honest answer is: yes, with qualifications. Several of them, actually. Computers are wonderfully cooperative until you ask a browser tab to download a neural network containing billions of parameters and then perform a considerable amount of linear algebra on your behalf.

The following points summarize my understanding of the key considerations and design aspects involved in developing this AI assistant:

1) The basic idea

The idea behind Chintan is surprisingly simple. The chatbot is embedded in a website, but the actual language generation happens locally inside the visitor’s browser, provided the device is compatible. A visitor types a question. The browser sends that question to the locally running model. The model generates the response, and the answer appears in the same browser. There is no need for every question to make a round trip to a conventional AI inference server. The model files are downloaded when they are first needed and can then be cached by the browser. This changes the economics of the application quite dramatically. There is no per-question API charge, there is no API key sitting inside a publicly accessible webpage waiting to be discovered, and the actual conversation can remain on the visitor’s device during inference.

There is an important qualification here, though. “Runs locally” does not mean that the webpage has magically become disconnected from the internet. The website and model host can still observe ordinary web activity such as page visits and model downloads. What happens locally is the interesting part: the visitor’s prompt and the resulting language generation do not have to be sent to a remote inference endpoint every time a question is asked. Chintan uses a JavaScript-based inference framework to load a quantized language model and execute it using WebGPU. Quantization is an important part of the story. Instead of representing every model parameter using relatively high-precision numbers, the model can use fewer bits per parameter. The result is a smaller model footprint and lower memory requirements, although this comes with some loss of numerical precision.

That is a trade-off I am quite happy to make. If the choice is between a slightly smaller model and asking someone to download several gigabytes of model weights before the chatbot says hello, the smaller model starts looking remarkably sensible.

2) The first challenge: loading is not thinking

One of the first things I discovered is that model loading and model inference are two completely different problems. Before Chintan can answer a question, the browser has quite a bit of work to do. It has to determine whether WebGPU is available, download the model if it is not already cached, initialise the model, prepare the execution environment, allocate the necessary memory, construct the prompt, and finally begin generating the response. So the first visit can feel surprisingly slow. This is not the assistant carefully contemplating the philosophical implications of your first greeting. It is mostly downloading and arranging a rather large collection of numbers. That distinction matters when designing the interface. Instead of hiding the preparation stage, Chintan makes it visible. The user can see that the model is loading and preparing itself, and the application can explain when the browser does not have the required capabilities. It may not be as visually elegant as pretending that everything happens instantly, but it is considerably more honest.

3) Streaming makes the wait feel shorter

A language model does not normally produce an entire paragraph in a single operation. It generates tokens sequentially, with each new token depending on what came before it. Chintan therefore streams the generated text into the chat interface as it becomes available rather than waiting for the entire response to finish. Technically, this does not make the model generate faster. The same computation still has to happen. But from the user's perspective, it makes a considerable difference. Watching an answer gradually appear feels much more responsive than staring at an empty chat box while the browser quietly works itself into a small existential crisis.

There is another small complication here. Language models can use special control tokens or internal structures while generating text. Those are useful to the model but are not particularly useful to the person reading the answer. A user-facing chatbot therefore needs to filter those artefacts and present only the intended response. Otherwise, the assistant can occasionally look as though it is muttering stage directions to itself. That is amusing exactly once.

4) Context is a finite resource

There is another limitation that becomes particularly obvious when running a model locally: the model does not have unlimited attention. Its context window has to accommodate the system instructions, recent conversation history, any reference material we decide to provide, and the response that is currently being generated. All of these compete for the same finite space. This means that simply increasing the maximum response length is not necessarily a good idea. If the model is allowed to generate thousands of tokens, more memory and computation are required, and less room may remain for the conversation itself. The solution is context management rather than simply turning every available number up to maximum.

Chintan keeps a bounded amount of recent conversation history, uses only the information that is relevant to the current interaction, and places a sensible limit on the response length. The aim is to give the model enough room to explain something properly without accidentally turning every answer into a small thesis.

5) Training is not the same as teaching

At this point, a natural question appears: what if I want Chintan to know about something specific? Perhaps laboratory protocols. Perhaps microbiome research. Perhaps course material. Perhaps a collection of documents or even my own blog posts.

The obvious answer is to train the model. But training is often not the best first solution. Fine-tuning can be useful when the goal is to teach a model a particular style, structure, behaviour or repeated task. It is much less convenient when the goal is simply to keep a body of factual information available and up to date. For that, retrieval is often a more sensible approach. The basic idea is straightforward. When a user asks a question, the system first finds the most relevant passages from a collection of documents. Those passages are then added to the prompt, and the local model generates an answer using the supplied information. This is commonly called retrieval-augmented generation, or RAG. I particularly like this approach because the knowledge base can be changed without retraining the model. If a document changes, update the document and rebuild the index. The assistant now has something new to read. It is considerably less dramatic than training a neural network from scratch, but drama is not normally a very useful unit of scientific progress.

6) What this approach does not solve

Running an AI model locally does not magically make it accurate, private in every possible sense, or suitable for every device. The approach depends on a reasonably modern browser with WebGPU support, sufficient GPU and system memory, a successful first-time model download, and a model that is small enough to run comfortably on the visitor’s hardware.

There is also the more fundamental issue of model capability. A smaller local model can be surprisingly useful, but it can also be confidently wrong. Moving the computation from a remote server to the visitor's computer does not somehow eliminate hallucinations. For scientific, medical, legal or otherwise high-stakes questions, Chintan should therefore be treated as a useful assistant rather than an oracle in a dark-green chat window. The answer may be a very good starting point, but important claims deserve verification.

So why did I build it?

Because I find the idea interesting. The dominant model of AI today is highly centralised. Large models live in enormous data centres, and users interact with them through remote inference servers. Local browser inference explores a complementary possibility: smaller models running much closer to the person using them, making use of hardware that is already sitting on their desk. Chintan is not an attempt to compete with frontier models on raw intelligence. That would be a rather ambitious use of a browser tab. Instead, I am interested in something more modest: an assistant that can be available without an API bill for every question, does not require me to maintain an inference server, and can perform its actual language generation directly on a compatible visitor’s device. It is a small experiment in a different way of building AI applications. And perhaps that is the most interesting part of it. We have spent years getting used to the idea that AI requires a giant machine somewhere else. It turns out that, for some kinds of AI, the machine might already be sitting in front of us.

Closing note: Chintan is an experimental on-device AI assistant. It doesn’t have a trillion-dollar data center hiding behind it - and it’s not pretending to. Instead, it explores just how capable AI can be when it runs locally in your browser. Have fun, learn things, experiment - and verify anything important.