Self-service, interactive analytics at multipetabyte scale in capital markets regulation on the cloud

Scott Donaldson and Matt Cardillo detail the security measures and system architecture needed to bring alive a multipetabyte data warehouse via interactive analytics and directed graphs from several trillions of market events, using HBase, EMR, Hive, Redshift, and S3 technologies in a cost-efficient manner.

FINRA is the largest independent regulator for all securities firms doing business in the United States. FINRA writes and enforces rules governing the activities of more than 4,035 securities firms with approximately 638,020 brokers. FINRA identifies market manipulations and rule violations by reviewing billions of market events daily, covering equities, options, and fixed-income markets, including the NYSE, Nasdaq, BATS, Direct Edge, and CBOE exchanges. Scott Donaldson and Matt Cardillo detail FINRA’s approach for innovating a secure, big data analytics solution to a critical business problem requiring interactive search capability for a multipetabyte data lake that increases by up to 75 billion of transactions each trading day. FINRA developed a breakthrough analytic solution for searching market events, generating order-lifecycle assemblies, and performing other market reconstructions to illuminate manipulative behavior in the markets from trillions of market events. This solution integrates leading big data analytic engines with AWS cloud services to challenge the norms of the IT reporting paradigms of the past. Scott and Matt describe the security measures FINRA adopted to protect financial records and detail the system architecture, which blends HBase for constructing order graphs from trillions of nodes and edges as well as providing slices of correlated market events from multipetabyte data lakes in the form of private data marts for interactive exploration and analysis.