How does Blue Coat WebPulse work with Blue Coat WebFilter (BCWF)?
What happens behind the scenes for a web site to be categorized dynamically?

The Dynamic Real-time Rating Service (DRTR) is one part of WebPulse that includes both foreground and background processes:

1. SGOS checks the current webfilter database (BCWF)  for a category rating. If no category rating is found, the WebPulse service using the DRTR attempts a category rating. The WebPulse service checks to see if a cached entry of the requested URL exists on the local ProxySG. If a cached URL exists, the rating is returned and no other activity occurs.

NOTE:  Both the above operations are conducted on the local (customer-owned) ProxySG.

Meanwhile, in the WebPulse cloud, the following occurs:

1. If no category rating can be found, WebPulse seeks a rating from one of the four data centers situated at strategic locations around the world. (Please see TECH242964 for the IP addresses of these four data centers.) 
2. When the WebPulse request is received in the datacenter, the receiving ProxySG appliance also checks it's own WebPulse cache to see if the website has already been categorized. If the answer is still "NO", then that ProxySG contacts the WebPulse rating service point IP address. 
   Note:  This service point IP address, is not advertised in any ProxySG appliance; it is part of the Webpulse cloud service.
3. The WebPulse rating service point checks its database for a rating and returns one immediately, if possible. If it does not have a rating for the website, it requests a rating to occur dynamically.
   Note: In total, at least three appliances running the Webpulse service ( DRTR) have checked their cached ratings database, before a dynamic rating occurs.

NOTE:  Due to the large number of WebPulse rating requests received from all ProxySG appliances deployed throughout the world, the database that the rating server keeps is usually slightly more up-to-date than the database on any particular ProxySG appliance. Both the service point and the WebPulse servers are different servers that operate together to deliver category ratings to the ProxySG appliance.

After a Successful Category Rating has occurred:

1. The service point and the ProxySG appliance both store the category rating in their caches and serve it to any other ProxySG appliance making the request. The main BCWF database is not updated with every dynamically rated URL. Based upon the URL's popularity and Blue Coat's ability to correctly categorize, the main ratings database gets updated. 
   Note: When dynamic ratings cannot occur, a request is sent for the URL to be categorized manually. The most popular requests for manual ratings are downloaded to Blue Coat's analysts (manual, human intervention) everyday, for analysis.

NOTE: With the K9 and ProxySG clients the behind-the-scenes actions are the same. Both the ProxySG and the K9 clients are told to keep the ratings for 24 hours.

Highlights of WebPulse:

• WebPulse has the primary focus on web threat detection using background processes. Uniting 75M users together into a collaborative defense with tremendous awareness to new web content and links is the main architecture benefit. Real-time inputs to WebPulse operations centers is the key element in the design because it greatly reduces response time.
• WebPulse today uses over 16 threat defenses including anti-malware, anti-virus, script analyzers, sand boxing, web hunters, machine token analysis and human raters.
• DRTR categorizes in 17 languages today for popular categories. WebFilter has 86 categories (times 21 languages) equals 1,360 possibilities. About 300 of these have been fully automated with machine token analysis, which is a huge amount of automation with 100,000s of rules and reference databases. (Compare this to 1800 rules on-box from Websense and one can see that DRTR is much larger in scope, however not 100% for 50 languages and 80 categories. The same can be said with Bright Cloud on Ironport, a simple on-box 'rule base' that is much smaller in scope than DRTR).
• Customers that have used DRTR over several years note that it provides a 4-6% higher categorization rate than not when not using it. In general, the BCWF database provides an 88% categorization rate. Adding DRTR to the process increases the categorization rate to between 94-96%.