An order book is the method of currency exchange used in traditional markets. In DEXs, an order book method is used to allow users to transfer assets in a decentralized manner. It a list of open buy and sell offers made by users for a specific number of assets. Every buy order needs to be matched with a sell order.
For example, a user makes an offer to sell A amount of X assets for B amount of Y assets, which is posted to the order book. When another user who wants to swap B amount of Y assets for A amount of X assets comes along, they fulfill that order and the assets are swapped between the users.