Back to Blog
·7 min read

Integrating Local LLMs with Ollama into Laravel Applications

AI LLM Ollama Laravel

Cloud LLM APIs are convenient, but they raise cost, latency, and data-privacy concerns. For internal tools and privacy-sensitive applications, running models locally with Ollama is a compelling alternative. I recently integrated Ollama into an online exam system to power an AI chat assistant, and the results were excellent.

What Is Ollama?

Ollama is a lightweight runtime for running open-source language models locally. It exposes an HTTP API that is compatible with the OpenAI chat-completions format, which means you can swap between local and cloud models with minimal code changes.

Installing and Running a Model

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2
ollama serve

The server listens on http://localhost:11434 by default. Verify it works with a simple curl request.

Calling Ollama from Laravel

Laravel's HTTP client makes this straightforward. Create a small service class to encapsulate the call:

use Illuminate\Support\Facades\Http;

class OllamaService
{
    public function chat(array $messages): string
    {
        $response = Http::timeout(120)->post('http://localhost:11434/api/chat', [
            'model' => 'llama3.2',
            'messages' => $messages,
            'stream' => false,
        ]);

        return $response->json('message.content');
    }
}

Streaming Responses

For chat interfaces, streaming makes the experience feel instant. Ollama supports server-sent events. In Laravel, return a StreamedResponse and forward the chunks as they arrive. On the frontend, consume the stream with fetch and a ReadableStream reader.

Prompt Engineering and Context

Always send a system prompt that defines the assistant's role and constraints. For exam systems, include the exam rules and a strict instruction to refuse off-topic requests. Keep the context window under the model's limit by trimming older messages.

Production Considerations

  • Run Ollama on a GPU-backed machine for acceptable latency; CPU-only inference is slow for larger models.
  • Put a queue in front of inference requests to avoid overloading the GPU.
  • Log prompts and responses for auditing, but scrub personal data.
  • Set timeouts and fall back gracefully if the model is unavailable.

Conclusion

Local LLMs with Ollama give you AI capabilities without vendor lock-in or data leaving your infrastructure. For internal tools and privacy-sensitive products, it is the best option available today.

All Articles