Job recommendations leveraging deep learning using Analytics Zoo on Apache Spark and BigDL

Can the talent industry make the job search/match more relevant and personalized for a candidate by leveraging deep learning techniques? Guoqiong Song, Wenjing Zhan, and Jacob Eisinger demonstrate how to leverage distributed deep learning framework BigDL on Apache Spark to predict a candidates probability of applying to specific jobs based on their rsum.

Collaborative filtering recommends items by identifying other users with similar taste but tends to misfire when user history is little known or new items are introduced into the mix. Incorporating context and natural language processing (NLP) is one way to improve recommendations. In addition, newly developed deep neural networks have shed light on the success by chaptering nonlinear relationships in the user-item dataset. In the talent attraction industry, short hire cycles limit history around job advertisements and job seekers. The implication is most job recommendation systems search via keywords. Unfortunately, this short keyword context lacks the expressiveness to adequately describe the job seeker’s intent. In contrast, résumés offer a source of much richer context in natural language. Guoqiong Song, Wenjing Zhan, and Jacob Eisinger demonstrate how to leverage distributed deep learning framework BigDL on Apache Spark to predict a candidate’s probability of applying to specific jobs based on their résumé, including document embedding using the pretrained Global Vectors for Word Representation (GloVe) model and neural collaborative filtering using deep neural networks. The deep learning algorithms in BigDL result in much better results compared to cosine similarity measure or traditional ALS (alternative linear square) as measured by precision and recall metrics.