Eventstore Replicator with grpc as a Reader is not working

Hello,

After a couple of years where I managed to always make it work with tcp as a reader, I am now stuck again.

With the recent release of 24.x tcp has been deprecated, I am currently in the process to migrate from 23.x to 24.x.

But for development/debugging it is still useful for me, from time to time to download a backup of the cloud.I have been using the replicator for that purpose, to replicate to a VM to which I have access in order to then download the backup.

I have been wondering if the issue with the grpc reader has been fixed, as in the documentation for the replication there is often the tcp used for the reader, and it is also “advised” to use the TCP: Known limitations | Replicator

Are there still issues with the grpc reader?

I’m a little confused by the title of this post. What exactly is not working? gRPC as a reader is available and works. When it comes to performance, one could say that any improvements in .NET and their support of gRPC will be reflected in EventStoreDB and Replicator. I’d advise you to try it out and use the built-in replicator metrics to see whether it satisfies your needs.

Hello,

thank you for the reply. I will try to expand.
I first noticed this issue and posted about it here a couple of years ago, where I could not get the replicator to work, if the source (reader) was using a gRPC protocol, and the solution was to switch to tcp for the reader.
In the limitations page, in the first paragraph it is said that it is recommended using tcp as a source and gRPC for the sink.

Now I am facing a similar issue where I am not able to make use the EventStore replicator, to replicate from a cloud eventstore instance to a vm on my gcp cloud. My source eventstore instance is on 24.2, hence tcp Is not supported. I tried using gRPC protocol for the source but the replicator just hung there doing nothing. So I had to restore a backup on an older (23.x) instance, in order to make the replicator work with tcp as a source.

Not sure if this explains better my issue.

That being said, from your answer, it seems like you are not aware of any issues in running the replicator with gRPC as both source and sink protocols. I will therefore try again and see if I missed something.

I will post back here once I get some more information, and I will try to get a reproducible example if the replication fails.

Thank you

TCP makes sense for older servers that only speak the TCP protocol.

I tried using gRPC protocol for the source but the replicator just hung there doing nothing

  • Do you have any logging associated with the replicator process when this happens?
  • Are you able to connect to the server/cluster using the gRPC protocol by any other means, e.g. a simple test script/program that uses the same connection string and one of our clients?

Hello,
sorry for not answering, we had a busy period.

For the logs, sadly no, I will try to get an example with a docker compose to see if I can give you a reproducible example.

Regarding being able to connect, I would say yes, as I have the actual application being connected to the instance in the source, and I am able to read fine.

Anyway, I will try to get an example when I have time, in the meantime, thank you for the answer.

Hello, I was in need to use the replicator again, but this time I was able to create a minimum viable example.

Initial problem:
I was trying to replicate from a eventstore cloud instance on a public network to my local machine deploying the replicator on docker. But since the cloud instance is on 24.6 I can not connect to it via TCP. (which for me was the only way I was ever able to run the replicator).
I tested with grpc and was not able to make it work.

So in order to try to replicate on a simple example I created a docker compose to replicate locally so that I could share it here. I looked at the various documentations but I couldn’t find something to unblock me.
I am able to connect to both eventstore instances via Admin UI at localhost:2113 and localhost:4113, so the instances are reachable. This is the same docker setup I use to connect with my local application with grpc client with the same connection string using your java client.
Here is the docker compose:

version: '3.7'

services:
  eventstore-source:
    image: eventstore/eventstore:24.6.0-alpha-arm64v8
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:2113/gossip" ]
      interval: 10s
      timeout: 10s
      retries: 3
      start_period: 5s
    environment:
      - EVENTSTORE_CLUSTER_SIZE=1
      - EVENTSTORE_RUN_PROJECTIONS=All
      - EVENTSTORE_START_STANDARD_PROJECTIONS=true
      - EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP=true
      - EVENTSTORE_INSECURE=true
      - EVENTSTORE_LOG_LEVEL=Verbose
    volumes:
      - './data-source:/var/lib/eventstore'
    ports:
      - 2113:2113
      - 1113:1113
  eventstore-target:
    image: eventstore/eventstore:24.6.0-alpha-arm64v8
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:2113/gossip" ]
      interval: 10s
      timeout: 10s
      retries: 3
      start_period: 5s
    environment:
      - EVENTSTORE_CLUSTER_SIZE=1
      - EVENTSTORE_RUN_PROJECTIONS=All
      - EVENTSTORE_START_STANDARD_PROJECTIONS=true
      - EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP=true
      - EVENTSTORE_INSECURE=true
      - EVENTSTORE_LOG_LEVEL=Verbose
    volumes:
      - './data-target:/var/lib/eventstore'
    ports:
      - 4113:2113
      - 3113:1113

  replicator:
    container_name: repl-replicator
    image: eventstore/replicator:0.4.8
    depends_on:
      eventstore-source:
        condition: service_healthy
      eventstore-target:
        condition: service_healthy
    ports:
      - "5000:5000"
    volumes:
      - './replicator.yml:/app/config/appsettings.yaml'
    environment:
      REPLICATOR_DEBUG: 1

And my replicator.yml file:

replicator:
  reader:
    connectionString: "esdb://localhost:2113?tls=false"
    protocol: grpc
  sink:
    connectionString: "esdb://localhost:4113?tls=false"
    protocol: grpc
    partitionCount: 1
  scavenge: true
  checkpoint:
    path: "./checkpoint"

And this is the error message I get from the replicator:

"Starting gRPC reader","@i":"59fda7be","SourceContext":"EventStore.Replicator.Esdb.Grpc.GrpcEventReader"}
2025-02-06T16:34:37.964733718Z {"@t":"2025-02-06T16:34:37.9589695Z","@m":"Reader error","@i":"450d20cc","@l":"Error","@x":"Grpc.Core.RpcException: Status(StatusCode=\"Unavailable\", Detail=\"Error starting gRPC call. HttpRequestException: Connection refused (localhost:2113) SocketException: Connection refused\", DebugException=\"System.Net.Http.HttpRequestException: Connection refused (localhost:2113)\")\n ---> System.Net.Http.HttpRequestException: Connection refused (localhost:2113)\n ---> System.Net.Sockets.SocketException (111): Connection refused\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)\n   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\n   --- End of inner exception stack trace ---\n   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)\n   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\n   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\n   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)\n   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)\n   --- End of inner exception stack trace ---\n   at EventStore.Client.Interceptors.TypedExceptionInterceptor.<AsyncUnaryCall>b__5_0[TRequest,TResponse](Task`1 t)\n   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()\n   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)\n--- End of stack trace from previous location ---\n   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)\n   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)\n--- End of stack trace from previous location ---\n   at EventStore.Client.GrpcServerCapabilitiesClient.GetAsync(CallInvoker callInvoker, CancellationToken cancellationToken)\n   at EventStore.Client.EventStoreClientBase.GetChannelInfoExpensive(ReconnectionRequired reconnectionRequired, Action`1 onReconnectionRequired, IChannelSelector channelSelector, CancellationToken cancellationToken)\n   at EventStore.Client.SharingProvider`2.FillBoxAsync(TaskCompletionSource`1 box, TInput input)\n   at EventStore.Client.TaskExtensions.WithCancellation[T](Task`1 task, CancellationToken cancellationToken)\n   at EventStore.Client.EventStoreClientBase.GetChannelInfo(CancellationToken cancellationToken)\n   at EventStore.Client.EventStoreClient.<>c__DisplayClass25_0.<<ReadAllAsync>b__0>d.MoveNext()\n--- End of stack trace from previous location ---\n   at EventStore.Client.EventStoreClient.ReadAllStreamResult.<>c__DisplayClass9_0.<<-ctor>g__PumpMessages|0>d.MoveNext()\n--- End of stack trace from previous location ---\n   at System.Threading.Channels.AsyncOperation`1.GetResult(Int16 token)\n   at System.Threading.Channels.ChannelReader`1.ReadAllAsync(CancellationToken cancellationToken)+MoveNext()\n   at EventStore.Client.EventStoreClient.ReadAllStreamResult.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()\n   at EventStore.Client.EventStoreClient.ReadAllStreamResult.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()\n   at EventStore.Client.EventStoreClient.ReadAllStreamResult.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()\n   at Ubiquitous.Metrics.Metrics.MeasureValueTask[T](Func`1 action, IHistogramMetric metric, ICountMetric errorCount, String[] labels, Int32 count)\n   at EventStore.Replicator.Esdb.Grpc.GrpcEventReader.ReadEvents(Position fromPosition, Func`2 next, CancellationToken cancellationToken) in /app/src/EventStore.Replicator.Esdb.Grpc/GrpcEventReader.cs:line 63\n   at EventStore.Replicator.Read.ReaderPipe.<>c__DisplayClass1_0.<<-ctor>g__Reader|1>d.MoveNext() in /app/src/EventStore.Replicator/Read/ReaderPipe.cs:line 41","SourceContext":"EventStore.Replicator.Read.ReaderPipe"}
2025-02-06T16:34:37.964869718Z {"@t":"2025-02-06T16:34:37.9645543Z","@m":"Reader stopped","@i":"49a9146f","SourceContext":"EventStore.Replicator.Read.ReaderPipe"}
2025-02-06T16:34:37.964876343Z {"@t":"2025-02-06T16:34:37.9646955Z","@m":"Storing the last known checkpoint","@i":"228021e9","SourceContext":"EventStore.Replicator.Replicator"}

Thank you in advance for any help

So after a night sleep I figured it out.

In my example the problem was the networking, I was trying to connect to localhost, but from within the compose I needed to use the service address, so it was supposed to be like this:

replicator:
  reader:
    connectionString: "esdb://eventstore-source:2113?tls=false&keepAliveTimeout=10000&keepAliveInterval=10000"
    protocol: grpc
  sink:
    connectionString: "esdb://eventstore-target:2113?tls=false&keepAliveTimeout=10000&keepAliveInterval=10000"
    protocol: grpc
    partitionCount: 1
  scavenge: true
  checkpoint:
    path: "./checkpoint"

The same was in the case of my connection to the cloud instance, the issue was that I was trying to connect to a evenstore instance running inside another docker compose, using localhost, but that didn’t work, so I had to include a evenstore db instance in the replicator compose and reference it by service (same as I did in the provided example).

It was helpful for me to use your Ask AI, helped me out quite a bit.

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.