Since I'm using a third-party database, I can't fix their indexing issues directly. I need to manually control pagination, so I'm wondering if there's a way to avoid retrieving duplicate indexes. Using SQL deduplication isn't efficient due to the large volume of data, making it difficult to fetch results effectively.

Now my problem is that during data migration, duplicate data appears in the queries. Since pagination is manually controlled, I have to perform deduplication after migrating the data to the target database. Because the data volume is large, this is very time-consuming. Is there a good solution?

I'm not quite clear, could you please elaborate?