Using AI locally

In most cases you run AI using a service provider. It’s easy and fast, and lets you access the latest models, which are typically trade secrets. However, there are two potential problems with this model:

• Cost. If you want to ask a few questions, most AI service providers will be happy to let you do that for free as a loss leader. But as you expand your AI usage, eventually they’ll expect to be paid.

• Privacy. The questions you ask, the prompts you provide, can teach somebody else a lot about what you are doing. Imagine this prompt: I have this brilliant trading strategy, which will make me millions as long as nobody else copies it, can you write me an agent to implement it?

The other option is to run an AI model locally. It is likely to be a lot slower, especially if you don’t have a GPU ↗, but it is possible. Read on for step-by-step directions.

Note

I am going to assume you’re using Visual Studio Code ↗, as set up in the user interface article.

Ollama setup

Ollama ↗ actually runs the LLM models.

1. Download Ollama ↗ and install it.

2. Download a model. Open a command line window and run this command.

   Tip

   To get to the command line on Windows, press Win+R and type powershell.

   ollama pull gpt-oss:20b

3. Run the Ollama UI, select the gpt-oss:20b model, and give it a simple prompt as a sanity check.

   Create a “hello, world” web page and save it as hello-world.html.

   Because you are running the model locally, it will take a long time to get the answer; that is normal. On my system getting the response took about four minutes. And because Ollama only runs models, it is not able to save it.

Visual Studio Code setup

To actually use Ollama from our development environment, we integrate it with Visua Studio Code using the Cline extension ↗.

1. Start Visual Studio Code.

2. In the right sidebar click the extensions icon.

   

3. Install the extension Cline. Click Trust Publisher & Install.

4. Click the Cline icon.

   

5. Click the gear icon in the top right of the left sidebar to set Cline up. You might need to create a free account.

6. Specify these parameters.

   Parameter	Value
   API Provider	Ollama
   Model	gpt-oss:20b
   Model Context Window	32768
   Request timeout (ms)	36000000

   Ten hours for a request timeout is excessive, but models running locally are a lot slower, especially if you don’t have a GPU.

7. Click Done. If there is a message about Git, ignore it. Git ↗ is an extremely useful tool, but not necessary at this point.

8. Give Cline a prompt.

   Create a “hello, world” web page and save it as `hello-world.html`.

   Why the ` ?

   By convention, in markdown text, such as AI agent directions, when a string is a name to be typed, such as a file name, it is enclosed in reverse quotes (`). Usually, you can find them at the top left of your keyboard.

   Wait until the model finishes figuring out how to do it. While waiting, you may need to click Resume or permit various actions.

9. See that in the root of your repository (the folder you opened with Visual Studio Code) you now have a page, hello-world.html.

Using AI in local mode

It proved too difficult to replicate the 3D tic-tac-toe game, so we’ll do something simpler: tic-tac-toe with a 4x4 playing board and a timer.

To expedite things, expand Auto-approve and approve, in addition to the defaults:

• Edit project files
• Execute safe commands
• Use the browser
• Enable notifications

Note that sometimes Cline works in two stages. First it creates a plan, and displays it for you. Then it waits for approval before execution. So when it appears stuck, tell it to continue.

Starting game

As before, start with normal tic-tac-toe.

Create a web page that lets two players play tic-tac-toe. Place everything in a single file.

Chat response

Tic‑Tac‑Toe game has been successfully implemented in a single HTML file: tic_tac_toe.html. Open this file in any modern web browser to play.

This is the result

Change board size

There was probably a tic-tac-toe game in the training input. To determine whether this model can create a new piece of software, we need to modify something.

Change the game board to be 4x4, with the same rules.

Chat response

Implemented a 4x4 Tic‑Tac‑Toe game in tic_tac_toe.html. Open the file in any modern browser to play.

This is the result

Add a timer

Add a five second timer. If a player does not move within the allotted time, move for that player into a random cell.

Chat response

Implemented a 4×4 Tic‑Tac‑Toe game with a five‑second turn timer that automatically places a mark in a random empty cell if the player does not move.

This is the result

A hidden timer makes for a terrible user experience. Let’s improve that.

Include a graphical representation of the timer.

Chat response

Added a visual countdown timer that displays remaining seconds and a green bar that shrinks over the 5‑second interval. The game auto‑moves if time expires.

This is the result

Final touchups

Some cosmetic changes to make the game more usable.

Several changes:

1. Make the progress bar smoother. Instead of removing 20% each second, remove 1% each 50 millisecond.

2. Show whose turn it is

Chat response

Implemented a smooth 5‑second countdown that updates every 50 ms and displays the current player beside the timer text.

This is the result

Conclusion

Whatever computer you use, it is probably going to be slower than the purpose-built machines that power the best models you can find on the internet. Whatever model you download, it is probably not going to be as good as the best commercial model. However, depending on your use case, that may not matter. Sometimes cost and privacy are more important than speed or sophistication.

What can I write for you?

My speciality is writing tutorials and other forms of documentation that help people use new technologies. If you found this tutorial useful, and you need documentation written, I can help.