How do I filter a query of an array of strings with a limit in DynamoDB pagination?

Question

When combining Limit and Filter Expression in DynamoDB, the database first retrieves the first N items specified by Limit and then applies the Filter Expression to those items. It's important to note that Limit in DynamoDB operates differently compared to traditional SQL limits.

Could someone please help me understand a good approach for implementing pagination? I find DynamoDB's documentation difficult to understand and not user-friendly. If anyone has an answer, please share with me. I'm finding DynamoDB frustrating to work with.

  const params = {
    TableName: TableName,
    ProjectionExpression:
      "a, b, c, #rg, d, e, f",
    ExpressionAttributeNames: { "#rg": "region" },
    FilterExpression: 'contains(:includedIds, xyz)',
    ExpressionAttributeValues: {
      ":includedIds": Array.from(Ids),
    },
    Limit: limit,
  };

 // xyz is primaryKey of my table 

  if (exclusiveKey) {
    params.ExclusiveStartKey = {
      xyz: exclusiveKey,
    };
  }

 dbResponse = await docClient.scan(params).promise();


//  My above pagination approach isn't work well. The below method I'm currently using involves making database calls in a loop, which is inefficient and very bad approach ..........
    
      let allItems = [];

      do {
        dbResponse = await docClient.scan(scanParams).promise();
        allItems.push(...dbResponse.Items);
        scanParams.ExclusiveStartKey = dbResponse.LastEvaluatedKey;
      } while (allItems.length < searchLimit && dbResponse.LastEvaluatedKey);

      dbResponse.Items = allItems.slice(0, searchLimit);
    }

Answer 1

We love hearing feedback on our documentation, please use the link at the bottom of any page you feel needs improvement.

As for your use-case, a Scan with a filter is a highly discouraged access pattern, you should try to create an index that suits your access pattern more specifically that way you will get more performance and cost efficiency and not have an issue with Limit and Filter.

However, you can over-fetch, meaning you increase your limit so that you read as close to 4KB as possible, as that is the minimum read consumption (0.5 RCU). Then you can modify your limit on the client side to return to the user.