Repository navigation
Local setup failing when using Ollama #6
Description
Hello! I have been trying out the project, and I like the idea very much. Congrats!
Now to the issue:
I have been able to setup the ecosystem using OpenAI, now I'm trying to run it using the local setup (with Ollama), and I'm facing some issues when trying to interact with the chat. I have been able to use other functionalities, like login, saving webpage snapshots to the DB, and so on. Here's the traceback from the backend:
Traceback
Traceback (most recent call last):
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
yield
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_transports/default.py", line 236, in handle_request
resp = self._pool.handle_request(req)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
raise exc from None
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
response = connection.handle_request(
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
raise exc
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
stream = self._connect(request)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 122, in _connect
stream = self._network_backend.connect_tcp(**kwargs)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
with map_exceptions(exc_map):
File "/Users/my.user/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 61] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
return await self.app(scope, receive, send)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/middleware/cors.py", line 93, in __call__
await self.simple_response(scope, receive, send, request_headers=headers)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/middleware/cors.py", line 144, in simple_response
await self.app(scope, receive, send)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
raise exc
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
await app(scope, receive, sender)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/routing.py", line 715, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
raise exc
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
await app(scope, receive, sender)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2405, in run_sync_in_worker_thread
return await future
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 914, in run
result = context.run(func, *args)
File "/Users/my.user/Github/SurfSense/backend/server.py", line 108, in get_user_query_response
localreturn = index.local_search(query=sub_query, search_space=search_space)
File "/Users/my.user/Github/SurfSense/backend/HIndices.py", line 246, in local_search
top_summaries_compressed_docs = top_summaries_retreiver.invoke(query)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_core/retrievers.py", line 254, in invoke
raise e
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_core/retrievers.py", line 247, in invoke
result = self._get_relevant_documents(
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain/retrievers/contextual_compression.py", line 44, in _get_relevant_documents
docs = self.base_retriever.invoke(
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_core/retrievers.py", line 254, in invoke
raise e
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_core/retrievers.py", line 247, in invoke
result = self._get_relevant_documents(
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_core/vectorstores/base.py", line 1039, in _get_relevant_documents
docs = self.vectorstore.similarity_search(query, **self.search_kwargs)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_chroma/vectorstores.py", line 582, in similarity_search
docs_and_scores = self.similarity_search_with_score(
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_chroma/vectorstores.py", line 679, in similarity_search_with_score
query_embedding = self._embedding_function.embed_query(query)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_ollama/embeddings.py", line 164, in embed_query
return self.embed_documents([text])[0]
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_ollama/embeddings.py", line 159, in embed_documents
embedded_docs = self._client.embed(self.model, texts)["embeddings"]
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/ollama/_client.py", line 262, in embed
return self._request(
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/ollama/_client.py", line 70, in _request
response = self._client.request(method, url, **kwargs)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_client.py", line 837, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_client.py", line 926, in send
response = self._send_handling_auth(
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_client.py", line 954, in _send_handling_auth
response = self._send_handling_redirects(
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_client.py", line 991, in _send_handling_redirects
response = self._send_single_request(request)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_client.py", line 1027, in _send_single_request
response = transport.handle_request(request)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_transports/default.py", line 235, in handle_request
with map_httpcore_exceptions():
File "/Users/my.user/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 61] Connection refused
What can be the issue? I believe the issue is closely tide to the Ollama implementation, the only difference between my working example with OpenAI is that I'm setting IS_LOCAL_SETUP= 'true'. Let me know if you'd like more details. Running from a Mac M2 (ARM64) chip.
Activity
MODSetter commentedon Sep 25, 2024
Hi @Coqueiro
Thanks for your kind words means a lot.
After looking at your Tracelog I believe its a issue with your running Ollama instance. By default SurfSense works on 'mistral-nemo' and ollama's default base url ....... you might have these different in your local.
You can change the Ollama Base URL and Model at these lines :-
INFO:
base_urlis param for base url https://python.langchain.com/api_reference/ollama/llms/langchain_ollama.llms.OllamaLLM.htmlSurfSense/backend/server.py
Line 61 in e991a2e
SurfSense/backend/server.py
Line 62 in e991a2e
SurfSense/backend/HIndices.py
Line 39 in e991a2e
If you change the embeddings them make sure to delete 'documents' table in PostGres and then delete verctorstores folder.
LMK if you still face issues
Coqueiro commentedon Sep 26, 2024
I think I understand my issue then: I wasn't running Ollama locally, I was under the impression this would be managed by the backend code.
Then, to use this setup I would need to run Ollama (probably following those instructions here: https://github.com/ollama/ollama?tab=readme-ov-file#ollama), and use the corresponding endpoint as a
base_url. Is that it?MODSetter commentedon Sep 26, 2024
You can skip base_url if Ollama is running at their default port :)