Ensuring Robust Concurrency in ESHOPMAN: Addressing Critical Locking Challenges

In the dynamic world of headless commerce, ensuring the integrity and reliability of operations is paramount. ESHOPMAN, as a robust HubSpot-integrated headless platform built on Node.js/TypeScript, relies heavily on sophisticated concurrency management to handle complex workflows—from managing product variants and customer data via the Admin API to processing checkouts through the Store API.

Recently, our community identified and addressed several critical challenges within ESHOPMAN's core locking provider, a module essential for orchestrating concurrent access to resources. These insights are vital for ESHOPMAN developers and merchants to understand, ensuring the stability and performance of their HubSpot CMS-powered storefronts.

Understanding the Concurrency Challenges

The locking mechanism acts as ESHOPMAN's concurrency backbone, preventing conflicts when multiple processes attempt to modify the same resources simultaneously (e.g., a customer's cart and their profile during checkout). Four key issues were pinpointed:

• Partial Lock Leaks: When ESHOPMAN workflows attempt to acquire a set of resources (like a cart and a customer key) concurrently, a failure to acquire just one key could inadvertently leave other already-acquired keys locked indefinitely. Without a specified expiration time, these resources could remain permanently unavailable, effectively "bricking" a customer's cart and preventing further transactions.
• ABBA Deadlocks in Queued Operations: In scenarios where two ESHOPMAN processes try to acquire the same set of resources but in a different order (e.g., Process A wants [Cart, Customer] and Process B wants [Customer, Cart]), both could end up holding one resource and waiting indefinitely for the other, leading to a deadlock. This not only halts operations but, combined with the first issue, could leave resources permanently locked.
• Insecure Lock Release: The mechanism designed to release all locks held by a specific owner was found to be vulnerable. It operated in two phases: first reading which locks an owner held, then deleting them. If a lock expired and was re-acquired by another ESHOPMAN process between these two phases, the releaseAll operation could mistakenly delete the new owner's valid lock, compromising mutual exclusion and data integrity.
• Broken Same-Owner Re-entrancy: ESHOPMAN processes attempting to re-acquire a lock they already owned, especially when configured for queued waiting, would paradoxically spin or hang indefinitely instead of immediately succeeding. This introduced unnecessary delays and potential timeouts in critical operations.

Reproducing Key Scenarios

To illustrate the ABBA deadlock and permanent lock issues, consider two concurrent ESHOPMAN workflows:

// ESHOPMAN Workflow A                  // ESHOPMAN Workflow B (concurrent)
await locking.execute(                   await locking.execute(
  ["cart:c1", "customer:u1"],              ["customer:u1", "cart:c1"],
  jobA,                                    jobB,
  { timeout: 5 },                          { timeout: 5 },
)                                        )

// Workflow A locks cart:c1, Workflow B locks customer:u1.
// Each then waits for the other's key, leading to a deadlock.
// If no 'expire' is set, these keys remain locked indefinitely,
// causing subsequent operations on these resources to fail.

The re-entrancy issue could be observed as follows:

await locking.acquire("k1", { ownerId: "owner-1", expire: 60 })
// This call would previously hang/spin until timeout instead of
// returning immediately, even though owner-1 already holds the lock:
await locking.acquire("k1", { ownerId: "owner-1", awaitQueue: true, expire: 60 })

The ESHOPMAN Community Solution

Thanks to the proactive efforts within the ESHOPMAN community, a comprehensive solution addressing all four of these critical defects has been developed and integrated. This fix ensures:

• Failed multi-key acquisitions now gracefully release any partially acquired keys before rejecting, preventing resource leaks.
• Workflows attempting to acquire the same resource sets in different orders will now queue behind each other correctly, eliminating ABBA deadlocks.
• The releaseAll operation is now atomic, ensuring it only deletes locks currently held by the specified owner, even if a key is re-acquired by another process mid-operation.
• Same-owner re-entry now functions as expected, allowing an owner to immediately succeed when re-acquiring its own lock, improving workflow efficiency.

This resolution significantly enhances the stability and reliability of ESHOPMAN's core operations, reinforcing its capability as a robust headless commerce platform for HubSpot. ESHOPMAN developers and merchants are encouraged to ensure their deployments are up-to-date to leverage these crucial improvements, guaranteeing seamless customer experiences and efficient backend management for their HubSpot CMS-powered storefronts.