Introducing WSO2 API Manager’s Throttling Implementation Architecture

Introduction

Previous versions of WSO2 API Manager used legacy throttling implementation that was subject to several drastic changes and restructure in the past few years. The most recent implementation of the legacy throttling implementation is based on Hazelcast atomic counters that sync the local counters of each gateway to sum-up clusterwide global count. Since it’s required to build a more complex and extendible throttling mechanism, we have decided to move forward with a Siddhi runtime-based throttling engine. Siddhi is a very powerful real-time event processing engine. With the new throttling implementation, we were able to overcome many of the limitations that existed in the old throttle implementation. This implementation adds one change to a typical API manager deployment where it’s required to have an additional instance to process data of every API request and take throttling decisions based on the applicability of available throttle policies. Throttling decisions are made by Siddhi runtime and throttling decisions are published to the JMS topic; each and every API gateway subscribed to this JMS topic throttling decisions get instantly notified to the API gateways.

Different types of throttling policies and level of applicability

WSO2 API Manager has several types of throttling levels. Each level has its own set of throttle policies. Applicability of throttle policies depends on the data of an incoming API request and the metadata of the API that belongs to the incoming request. Users may apply one or more throttling policies at different levels based on their requirement. Following are throttling policy applicability levels:

• Application Level
• Throttle policies defined at this level are applied based on the application details of an API that made the subscription. These policies will limit API access considering the subscribed application details of an API. The purpose of having policies at this level is to control API access by considering the end user of an application.

• Subscription Level
• Throttle policies defined at this level are applied based on the subscription details of a API and the application details of the application that’s going to subscribe an API. These policies will limit the total number of requests that can go through from an application to a single API.
• API Level
• Throttle policies defined at this level are applied based on the metadata of the API that belong to a particular API request. The purpose of having policies at this level is to limit the total number of requests coming to an API from any number of users from any number of applications.
• Resource Level
• Throttle policies defined at this level are applied based on the resources available in a particular API and its metadata. Users may apply policies at this level to give different throttle limits to resources, e.g. a user may grant 20 req/s for GET method of a API and 1 req/s for DELETE resource of an API.
• Global Level
• Policies applicable at this level will be applied to all APIs of a gateway. If a user deployed a global level throttle policy then it will be applied to any API available in the system.

There are several policy types that are applicable in the above levels. Those policies can be categorized as follow:

• Per Token Quota Policies
• These policies are applicable at application level. The user can specify per token quota policy when he/she is creating or editing an application from the store. The limit of a policy specifies the maximum number of request or bandwidth that the end user can access when considering all subscribed APIs of a single application. For example, having per token quota tier of 10 req/s means that each end user of the particular application can invoke any API that’s subscribed to the application at only 10 req/m.

Figure 1: Add application page

• Subscription Level Policies and Burst Control
• As the name implies, these policies are applied at subscription level. As in Figure 2, the user needs to select a subscription level policy when the API is subscribed to an application. The purpose of a subscription level policy is to limit the number of request flown from an application to a single API. The limit of this policy will be the maximum number of requests or bandwidth that any number of application users can make for a single API through an application. Burst control is another feature that’s a part of subscription level policies. As in Figure 3, the burst control limit can be defined in the subscription policy view. The functionality of burst controlling is to control request bursts that come in a very short time to an API from a single application. Moreover, a real scenario would be when the user subscribes to a policy that has a very large request limit for larger time intervals, such as days or months. So it’s possible to consume all the allocated quota within a short period of time even if the policy is expanded for larger time intervals. With burst control, the person who defines subscription policies can further define a maximum rate per a second or a minute that controls the burst of request that comes to an API from a single application.

                                          Figure 2 : Subscription Policy Selection

                               

Figure 3: View of Defining Burst Control Limit in a Subscription Policy

                

• API Level and Resource Level Policies (Advance Throttling Policies)

• Even though these two type of policies are applied at different levels, the policy definition will be done through a single window. These policies are a set of advanced policies in the API Manager that enable functionality to perform throttling based on IP, IP Ranges, Headers, Query Params and JWT Claims. As the name implies, API Level policies will be applied at the API Level while Resource Level policies will be applied at the API Resource Level. When the API is created, users can apply these advanced throttle policies only at API level or resource level. If a user selects applying policies at resource level policy, then the API Level policy selection will be disabled and vice versa.

                                   Figure 4: View of Selecting Policy Level in API Manage Phase

• Custom Policies
• Custom policies have been introduced in the API Manager 2.0.0 release. These type of policies will be applied globally across all APIs. Custom policies allow users to write custom Siddhi query. Depending on the data received from gateway API calls, these policies will be executed and will send throttling decisions through JMS topic to gateways. Users may write Siddhi queries to limit activities of certain users based on their API invocation data that comes to the traffic manager. The throttle key is an important part of defining the policy. Key template definition available in the policy configuration will be sent to the gateway and then replaced with actual values with the request and will decide if the current request is throttled or not. These policies extend the flexibility of throttling by allowing users to write custom throttle policies. The custom policy in Figure 5 will limit the number of API calls that can be made by an admin user for any API. Likewise, the user may write queries according to their requirements.

Figure 5: Sample Custom Policy

• Blacklist Conditions

• Blacklist conditions is also a new feature that was introduced with API Manager 2.0.0. Blacklist conditions provide functionality to instantly block a user who invokes APIs by username, blocks API calls coming from a specific IP address by specifying IP, blocks an API by name, and blocks API calls coming from an application by application name. As in Figure 6, these policies will be defined from the admin dashboard that will be instantly applied to all gateway nodes. Blacklist conditions will be applied globally for any API call coming to the API gateway. For example, if an administrative user notices suspicious behavior of an application user, he/she may block the user with the blacklist condition.

Figure 5: View of Defining Blacklist Conditions

Policy types discussed above are the main types of throttle policies available in the API Manager. Figure 6 shows the complete diagram of the applicability of above policies at different levels.

                                   Figure 6: Throttle Policy Applicability at Different Levels

Deployment Architecture

Deployment Architecture Overview

A typical API Manager deployment pattern will change slightly along with the changes introduced in the new throttling implementation. Given that it’s required to have a separate policy engine to process data coming from every API call, an API manager deployment will require to have an additional instance to do the throttle policy processing functions. This additional instance contains Siddhi runtime that will process real-time API data and execute the throttle policies. The main components of a typical deployment of an API Manager with the new throttling implementation are as follows:

• API Publisher and Admin Dashboard
• This node is typically used to publish APIs to the gateway. Most times admin dashboard also will be deployed in the same node. Users can define throttle policies through the admin dashboard that will be deployed in the node that contains the policy engine.

• API Store
• API Store node will display all available APIs from API publishers. Application developers will create applications and subscribe to the required APIs through the API Store node.

• API Gateway
• The responsibility of the API Gateway is to serve API requests that come to the gateway by applying security, throttling, etc. The gateway will publish usage statistics to API analytics servers if analytics has been switched on. With the new throttling, the gateway will extract the required data to perform throttling and publish to the traffic manager that makes the throttling decisions. The throttling decisions from the traffic manager comes through JMS Topic. Hence, the gateway contains a JMS listener to listen to a topic created in a message broker to receive throttling decisions.

• API Key Manager
• This node will validate the access token that comes with API call and sends its validity and required API metadata to the gateway. This node is also responsible for generating refresh tokens and expire tokens depend on user requests and access token validity.

• API Traffic Manager
• This is the newly added instance to do the throttle policy evaluation through Siddhi runtime. When the API request comes to a gateway, it will send data to the traffic manager node which will be processed by the traffic manager instance to take throttle decisions. The traffic manager will execute the throttle policies against data that comes with every event and take decisions based on applicability of each throttle policy available in the system. If a particular request is throttled, then the traffic manager will send those details to a JMS topic. Every gateway node is subscribed to this JMS topic, hence throttle decisions will be notified to the gateway through JMS messages.

Figure 7 shows the deployment diagram with the new throttling implementation. It shows the interactions between the nodes and databases. Some diagrams below are used to separate the message broker instance for clear demonstration. Typically, production deployment will use a message broker that’s shipped with the traffic manager instance.

        Figure 7: API Manage Deployment Overview

Throttle Data Publishing Functionality

The data required to take the throttle decision from the traffic manager needs to be published from each gateway node. It’s required to publish data from each and every request that comes to the gateway. Binary or thrift transport can be used to publish data from the gateway to the traffic manager. Data publishing has been fully implemented asynchronously in order to remove the overhead of data publishing from the API request path. Hence, the API request is unaffected by data publishing overhead. Even if the traffic manager instance goes down, the API gateway will continue to serve the API requests, but throttling won’t be applied. Figure 8 shows how data publishing has been done without affecting the API request.

                  Figure 8: Data Publishing To Traffic Manager

As in Figure 8, the throttle handler will extract the required data and pass it to the throttle data publisher that’s responsible for publishing throttle data asynchronously without affecting the API request. Message ID, Application Throttle Key, Application Tier, API Throttle Key, API Tier, Subscription Throttle Key, Subscription Tier, Resource Level Throttle Key, Resource Level Tier, End User, API Context, API Version, Tenant Domain of owner of Application, Tenant Domain of the owner of API, Application ID, Remote IP and API Name will be published for every request from the gateway to the traffic manager. If publishing request headers, and JWT Claims and query parameters are enabled, these data also will be published from every request that comes to the API gateway.

Throttle Policy Creation

Traffic manager instance makes throttle decisions based on throttle policies deployed in its environment. When a user creates a throttle policy from user interfaces of the admin application, the API manager generates a compatible policy in Siddhi query language. User interfaces that defines throttle policies hide the complexity of the Siddhi query language by providing a rich UI to configure throttle policies. The API manager uses a set of predefined templates to generate complex Siddhi query based on user inputs. The user can define throttle policies through the admin application. The admin application contains separate views to define advance throttling, subscription tiers, application tiers, custom rules, and blacklist conditions. Since blacklist conditions process through the gateway itself there won’t be any Siddhi query language based throttle policy generated when adding a blacklist condition. When a user adds a throttle policy, the API manager will generate a Siddhi query language based throttle policy and deploy it to the traffic manager. Figure 9 shows the flow when a user adds a throttle policy.

Figure 9  Publishing Throttle Policy To Traffic Manager

Throttle Policy Definition

The traffic manager instance is responsible for processing data comes through throttle stream and makes decisions. Once the traffic manager receives a request, Siddhi runtime will process the deployed policies in the traffic manager instance. Figure 10 shows different sections of a throttle policy.

• Policy name
• This section contains the name of the throttle policy
• Policy description
• This section contains the description of throttle policy
• Request Stream Definition
• This section defines the mapping between incoming data from throttle request stream to a Siddhi stream named RequestStream that can be understood by Siddhi runtime.
• Global Throttle Stream Definition
• This section defines the mapping of Siddhi output stream named GlobalThrottleStream to outgoing global throttle stream. This section has done the opposite mapping performance in the above section.
• Eligibility Query
• This section contains the eligibility query. This query will decide whether the policy will be further processed or not. Every the policy deployed in traffic manager will execute up to this point for every request. If the data from the request stream matches the criteria defined in the eligibility query then it will be further processed. For example, check throttle state section in the sample policy in Figure 10 will only be executed if the subscription tier is gold and the tenant domain of the API owner is carbon.super which is specified in the eligibility query.
• Check Throttle State
• This section contains the query that checks whether the current request is throttled or not. It performs a count operation by grouping the throttle key that comes from the stream. This query checks the current request within a specified time interval of policy and checks if the allowed request count has exceeded or not. This query outputs throttle key, expiry timestamp of current time window and throttle state whether the request is throttled or not.

• Export to Global Throttle Stream
• This section contains the operation that sends the throttle decision to the JMS Topic. Query in this section sends a message to the JMS Topic if there is a change in the throttle state. Moreover, if the current request is throttled, then the query will send a message to the JMS Topic. This is because if a new gateway spawn or old gateway becomes active, the throttle state will need to be received to the newly added gateways to the cluster. So when the first request comes to the gateway it will trigger the throttle policy in the traffic manager. Only if that request is throttled, the traffic manager will notify gateways. If the request is throttled, gateways won’t publish any data to the traffic manager until the throttling time interval is over.

Figure 10: Sample Throttle Policy

Throttle Decision Making and Notify Throttle Decision

The Traffic Manager evaluates throttle policies against each of the requests that comes from the gateway and makes throttle decisions. Since the Traffic Manager evaluates all policies deployed in its environment, Eligibility Query decides if the policies will be further processed. As mentioned in the throttle policy definition section, when a particular request meets with the condition specified in the throttle policy, it will trigger an extension that sends a throttle decision to the JMS topic. Gateways that subscribe to the JMS topic will get the throttle decisions from the traffic manager. Figure 11 shows the flow of how the traffic manager instance notifies throttle decisions to gateways through the JMS Topic. Throttle decision message that comes to the gateway contains throttle key, throttle window expiry time, and throttle state. These properties will be used by the throttle handler to take decisions on incoming messages to the gateway.

 Figure 11: Notify Throttle Decision

Custom Policy Feature

Custom policies provide the ability to define user-defined throttle policies. As described above, custom policies contains two major parts, namely throttle policy definition and key template. When writing a throttle policy it’s required to define a throttle key to maintain counters against it. In a custom policy it’s required to have the same throttle key in the throttle policy and key template. It’s because the gateway node will replace the key template value with actual values coming from the request and check whether the template key is present in the throttle data map. For example if an administrative user needs to limit the requests of a particular user, then he/she will select the key template to be $userId and the throttle key to be the user ID as well. If a policy is defined to limit a user called “testuser”, then the throttle key of the policy will be “testuser”. Once the policy is triggered, the throttle message will be sent from the traffic manager with the throttle key “testuser”. From the gateway $userId template a key will be replaced by an authenticated user of the incoming request and checked whether the key is present in a throttle data map. Figure 11 shows what happens when a custom policy is added to the deployment.

                  Figure 12: Deploying Custom Policy

As in Figure 12, when a policy is added or updated in from the admin dashboard, there will be three main operations that happens in the system. One operation is deploying the throttle policy into the traffic manager. Thereafter the key template will be published through the JMS topic to the gateway. The final operation is saving the custom policy with a key template into the database. When a new gateway instance is spawned, it needs to get all available key templates of the system. To get the all available key templates, the newly spawn gateway will call a key manager instance to retrieve all available key templates in the system. This way the gateway node will receive all available blacklist conditions in the system.

Blacklist condition Feature

Blacklist conditions allow admin users to block API invocation by API Name, API User, Application, and IP. These conditions will be instantly applied to all the gateways in a deployment. There won’t be any Siddhi query language based throttle policies deployed in the traffic manager to make throttle decisions for blacklist conditions. The decision based on blacklist conditions are only processed and taken within a gateway. Figure 12 shows operations performed when an admin user adds a blacklist condition. There won’t be any policy deployment in the traffic manager during the stage of adding blacklist conditions. When adding a blacklist condition, the condition type and condition value will be published to the JMS topic which will be received by all subscribed gateways. The blacklist conditions will be saved in the database as well.

Figure 13: Adding Blocking Condition

When a new gateway spawns it will fetch all available blacklist condition from the key manager initially. Then block condition updates will be fully communicated through JMS messages.

Throttle Decision Making at Throttle Handler

Deciding if the incoming request is throttled or not is a critical functionality of an API gateway. The API manager has a dedicated handler to take the throttling decisions based on incoming request data. The above sections discussed how throttle decisions comes to an API gateway. New throttling implementation contains several data maps to keep throttling decisions. Throttling decisions made by the traffic manager will be inserted into a single data map. Key templates of custom policies and blacklist conditions are separately maintained in different data maps.

Figure  13: Taking The Throttle Decision

When a request comes to a gateway, the throttle handler first checks whether any blacklist conditions are present in the data maps. If blacklist conditions are present, it will evaluate the request data against the present blacklist conditions. For example, if a admin user adds a blacklist condition to block an end user who invokes APIs, then it will be evaluated and a decision will be made by the throttle handler. Afterwards, the throttle handler creates the resource level throttle key, API level throttle key, subscription level throttle key, and hard limit throttle keys to evaluate and understand if the current request is throttled at any level. First, the handler will check resource level throttling and API level throttling by checking to see if the resource level throttle key or API level throttle key is present in the throttle data map. If present, it will further check the expiry time window of the throttle decision and decide if the current request is throttled or not. If the request isn’t throttled at these two levels, then subscription level throttling will be applied. When taking the subscription level throttling decision, the handler will first check if subscription key is present in the throttle data map. If not, burst controlling will be applied to control request bursts. If the request isn’t throttled at this level as well a per token quota based throttling will be applied. If per token quota throttling is also not applied then a handler will check whether any key templates are present in the key template data map. If key templates are present then a handler will replace template values with actual values and evaluate them against with data maps to check whether the current request is throttled or not. Finally hard throttling will be applied to protect the API backend. If a request doesn’t get throttled out and any level, it will be sent to the backend. The format of throttled out messages received for each level of throttling is as follows:

Api Level Throttled Out Response

{
 "fault": {
   "code": 900800,
   "message": "Message throttled out",
   "description": "You have exceeded your quota",
   "nextAccessTime": "2016-May-21 15:41:00+0000 UTC"
 }
}

Hard Limit Throttled Out Response

{
 "fault": {
   "code": 900801,
   "message": "API Limit Reached",
   "description": "API not accepting requests"
 }
}

Resource Level Throttled Out Response

{

"fault": {
   "code": 900802,
   "message": "Message throttled out",
   "description": "You have exceeded your quota",
   "nextAccessTime": "2016-May-21 15:05:00+0000 UTC"
 }
}

Per Token Quota Throttled Out Response

{
 "fault": {
   "code": 900803,
   "message": "Message throttled out",
   "description": "You have exceeded your quota",
   "nextAccessTime": "2016-May-21 15:36:00+0000 UTC"
 }
}

Subscription Level Throttled Out Response

{

"fault": {
   "code": 900804,
   "message": "Message throttled out",
   "description": "You have exceeded your quota",
   "nextAccessTime": "2016-May-21 15:36:00+0000 UTC"
 }
}

Blocked Response

{

"fault": {
   "code": 900805,
   "message": "Message blocked",
   "description": "You have been blocked from accessing the resource"
 }
}

Throttle Out by Custom Policy Response

{
 "fault": {
   "code": 900806,
   "message": "Message throttled out",
   "description": "You have exceeded your quota",
   "nextAccessTime": "2016-May-21 17:05:10+0000 UTC"
 }
}

Summary

The new throttling implementation of WSO2 API Manager is very powerful and offers significant advantages compared to the old throttling implementation. This article discussed the architecture of new throttling implementation in detail. The new throttling implementation offers the flexibility to create complex policies, such as defining throttling policies considering the values of headers, query parameters, remote IP and JWT claims coming through a request. Moreover, custom policies allow users to create user-defined policies to meet their requirements. The newly added blacklist feature provides the ability to prevent attacks that come to the gateway by instantly blocking API calls by API Name, API Context, IP and application name. These functions highlight the power of the new throttling implementation.