Petabi REsolutions represents the umbrella product employing all of Petabi’s regular expression technologies. These include REconverge for universal event aggregation and evaluation, REmake for automatic signature generation, and REmatch for high-speed regular expression matching. REsolutions also includes a suite of machine learning tools to train REconverge and evolve the system. Finally, REsolutions includes the infrastructure to tie all the pieces together including report generation, management, and interconnectivity.
: Regular Language Processing of Events
Petabi has recognized that data of any kind can be used to create a detector in an information system. Further, these detectors all speak a language. That language describes the features that are recognized as meaningful to the system. We have developed both a framework and a tool, called REconverge, that will read these events and automatically condense multiple events into a single event that can be propagated further into the system. Since REconverge processes events as a language it is possible to examine event patterns across normal boundaries. There is no necessity that the event-generating detectors share a common application or source. REconverge can heterogeneously merge event streams from an arbitrary number of event generators and from that merger derive heretofore unidentifiable correlations between events across detectors and across time. This incredibly powerful feature is not hampered by the source of the event as it may be anything at all so long as it can produce a binary or textual output. With REconverge it is possible to describe relations and patterns between detectors that simply is not possible in modern systems.
Perhaps more importantly, REconverge utilizes regular languages to describe the event patterns that have more meaning within the system. There are numerous advantages to this approach. First, regular expressions provide users of the system a wide and dynamic syntax with which to describe correlations among events. Second, regular expressions can easily be added, updated, or removed from the system. Thus, changing to new circumstances is as simple as updating the set of regular expressions describing system-wide behaviors. Further, Petabi’s extremely fast Regular Expression matching engine, REmatch, ensures that processing of large sets of regular expressions is not only viable, but ultimately more efficient than other means. Regular expressions represent semi-dynamic state machines. Since REconverge recognizes event patterns as regular expressions this works as simulating the complex behaviors of event interactions. Since it is possible for the regular expression set to be updated then the system, as a whole, will learn over time, becoming smarter and more tailored to an individual organization. Combined with feedback, this enables one of the simplest and most dynamic machine-learning systems for modeling event behaviors. Finally, REconverge provides an abstraction to the actual data thus allowing analysis of event data to occur independent of data storage or archival. This means that data analysis using REconverge can happen distributed, across the system, or on top a database of aggregated data from another process depending on the desire of the system stakeholders. To REconverge, however, either system is just as effective.
: Automatic Signature Generation for Network Detection Signature Refinement
Petabi has developed REmake, a tool that employs a mutational (genetic algorithm) to automatically learn and create a variant resistant signature for identifying malicious exploits. REmake also optimizes these signatures to the locale ensuring the maximum efficiency of the signature during matching. This process works by simulating and mutating exploits and then deriving an optimized signature from a number of successive iterations. The benefit of REmake is that an organization can define when and where to craft signatures, can refine those signatures with local traffic samples, and even test the signatures and optimize them for efficiency to produce the most succinct set of regular expressions that can identify all desired exploits. This allows the organization to control their own signature set and manage it with minimal effort.
REmake can also process live samples as might be detected in a working system. These live samples can be fed into REmake and a signature will be produced that aligns with the invariant portions of those samples. Thus, a sampling of several dozen phishing email could be fed into the system to produce a signature by which those phishing attacks could be filtered. Samples may be binary or text. This allows for creating a feedback system such that as new threats are detected they can quickly and accurately be added to a filtering or protection system. Even more, the signature optimization can be used to identify signatures that might cause problems or harm the system and refuse to apply those as might occur if an attacker is able to inject samples designed to subvert the system. REmake makes keeping an organization’s signatures effective and up-to-date and as simple as a push of a button.
: PCRE-compatible High-speed Regular Expression Matching
REmatch is Petabi’s flagship regular expression matcher capable of multi-Gigabit throughput even when matching thousands of rules against an input stream. REmatch has implemented all of the features of Perl Compatible Regular Expressions (PCRE), including backreferences and assertions, while still meeting high-speed matching demands. This added power gives users not only high-speed matching, but also all of the versatility of PCRE making REmatch one of the first automata-based matching tools to meet such a high standard.
Regular expressions require considerably more effort to match than fixed string patterns but offer a corresponding amount of improved flexibility and dynamism absent in fixed string patterns. In order to match regular expressions at line-speed it is typically necessary to introduce specialized hardware. REmatch utilizes parallelism already present in modern CPUs as well as employing a matching automata optimized through our REduce technology, and other means, for high-speed matching. The result is a vast improvement in matching speed over traditional regular expression matching methods when matching multiple rules against an input. More importantly, REmatch is a library and does not require specific hardware solutions.
* Thousands of regular expressions matched at multi-Gigabit speed
* Small matching signature
* Simple API for incorporating into projects
* Full PCRE-compatibility
* On-line Rule Update
* Streaming or block mode
Regular Expression Matching Performance by Number of Rules for REmatch, PCRE, PCRE-JIT, Boost::regex, LLVM, and Hyperscan.
REmatch for Network Security:
Network Security often requires matching a set of patterns to network traffic in order to identify potentially malicious traffic. Regular Expressions offer a more expressive syntax for identifying patterns over fixed binary strings.
REmatch for Content Management
Identifying content tags across many files or finding a set of patterns in files is a common operation in modern content management and content search systems. REmatch can greatly improve performance and is easily included in current code utilizing a simple API.