Online coding assessments are the strongest tool hiring teams have to assess the soft and technical skills of a candidate. 

However, online coding tests are only effective if they are held to a high standard of integrity. When left unchecked, threats like cheating, content leaks, and impersonation can undermine the fairness and validity of these tests. 

At HackerRank, we understand the importance of maintaining a level playing field for all candidates, and we have implemented a robust set of measures to address these challenges and uphold assessment integrity.

Detecting Suspicious Spatial Environments

One of the primary concerns in online assessments is the use of physical surroundings to gain an unfair advantage. Candidates could attempt to cheat by seeking help from nearby individuals. Some might also resort to using unauthorized external devices such as webcams or additional monitors. To counter these threats, HackerRank employs a number of advanced proctoring techniques.

Multiple Monitor Detection

This feature detects if an external monitor is connected to the device on which the candidate is taking the assessment. By identifying potential sources of unauthorized information, we can ensure that candidates are focused solely on the task at hand and don’t receive external assistance.

Image Proctoring

During an assessment, HackerRank’s image proctoring feature records images of the candidate’s environment and displays them on the candidate’s report for review. This allows hiring teams to identify any suspicious activities, unauthorized materials, or additional individuals in the candidate’s surroundings.

Image Analysis

Complementing image proctoring, our advanced image analysis algorithms analyze the candidate’s environment and detect potential irregularities, such as multiple faces appearing on camera. This technology helps identify potential instances of impersonation or unauthorized assistance from nearby individuals.

Mitigating Impersonation Risks

Impersonation, where someone else takes the test for the candidate, can challenge the integrity of the assessment process. To address this risk, HackerRank employs identity verification measures to ensure the candidate is the only person taking the test.

Photo Identification

Before starting an assessment, HackerRank’s system captures a photo of the candidate. This image serves as a reference for verifying the candidate’s identity throughout the assessment process.

Image Proctoring & Analysis

In addition to capturing the candidate’s photo, our image proctoring feature captures the candidate’s image in 60 seconds intervals throughout the assessment. These images are then analyzed using facial recognition techniques to detect potential instances of impersonation or external assistance.

Identifying Suspicious Coding Behavior

In addition to monitoring the candidate and the physical environment, HackerRank also employs techniques to detect suspicious coding behavior during assessments. Candidates may attempt to gain an unfair advantage by using unauthorized resources like browser extensions or AI agents, copying solutions directly from other websites, or collaborating with another person on the assessment.

Copy-Paste Tracking

Our copy-paste tracking feature detects if a candidate pastes copied code from an external source during the assessment. This ensures that candidates are not relying on pre-written solutions obtained from unauthorized sources.

Tab-Switch Proctoring

By monitoring tab switches during an assessment, HackerRank can identify if a candidate is accessing unauthorized resources or seeking external assistance. This feature helps maintain a controlled testing environment.

AI-Powered Plagiarism Detection

To identify instances of plagiarism, HackerRank uses an advanced detection system that feeds proctoring signals and user-generated data into a supervised machine learning algorithm. Our plagiarism detection system can identify and flag suspicious behavior during an assessment, such as the use of browser extensions, AI agents, or external assistance.

This ML-based approach is three times more accurate at detecting plagiarism than traditional code similarity approaches and dramatically reduces the number of false positive plagiarism flags.

Preventing Test Content Leaks

Test content leaks pose a significant threat to the validity of assessments, as candidates may gain access to leaked solutions or question structures from online forums or developer communities. This provides an unfair advantage to test takers who have access to this information. To combat this issue, we use the following techniques:

Shuffled Sections & Questions

By presenting test sections and questions in a randomized order to each candidate, we mitigate the risk of content sharing and ensure that no candidates benefit from leaked information or online solutions tailored to specific question types or structures.

Watermarking

HackerRank’s watermarking feature adds the logged-in user’s email ID as a watermark to certain types of questions. This acts as a deterrent against candidates taking pictures and sharing question content, as the watermark can be used to trace the source of any leaked material.

Hiding Question Labels

To further prevent content sharing and online searches, HackerRank hides the titles of questions from candidates during assessments. This makes it more difficult for candidates to search for questions by using keywords, limiting the potential impact of leaked content.

AI-Solvable Content Label

Our library questions are labeled according to their potential solvability by AI assistants. Armed with this information, hiring teams can filter out questions that may be susceptible to AI-generated solutions. This ensures a more reliable assessment of a candidate’s skills and knowledge while mitigating the risk of content leaks facilitated by AI tools.

Key Takeaways

At HackerRank, we’re committed to maintaining the highest standards of assessment integrity. By combining advanced technologies with robust proctoring measures we’re able to identify and contain a range of potential threats to assessment integrity.

Upholding this high standard is an ongoing process, and HackerRank is committed to staying ahead of emerging risks by using the latest advancements in AI, machine learning, and proctoring technologies. By partnering with HackerRank, companies can confidently assess candidates’ skills and abilities, knowing that our rigorous integrity measures safeguard the validity and reliability of the assessment process.

Frequently Asked Questions

Can Your Plagiarism Detection System Detect Code From ChatGPT?

Yes. Our AI-enabled plagiarism detection system feeds several proctoring and user-generated signals into an advanced machine-learning algorithm to flag suspicious behavior during an assessment. By understanding code iterations made by the candidate, the model can detect if they copied and pasted code from an external source. However, it isn’t possible to identify what source the candidate used to obtain or create the code.

Does Your Plagiarism Detection System Automatically Fail Candidates?

No. Our detection system identifies potential cases of plagiarism and empowers hiring teams to decide if it’s an actual case of plagiarism.

I Still Have Questions About Plagiarism. Who Should I Contact?

If you’re a customer looking for support on plagiarism and its impact on your business, you can contact your customer success manager or our team at support@hackerrank.com.

The post Maintaining a Level Playing Field: HackerRank’s Commitment to Assessment Integrity appeared first on HackerRank Blog.

Being a SaaS platform admin isn’t easy. Defining the right level of user access can be challenging when you start factoring in numerous users and complex internal workflows. 

In addition, your platform may not give you enough customization to define what your users can and can’t do, which can lead to users being given a role with the wrong amount of access. 

Recognizing this need for flexibility and efficiency, we’re always working to make the lives of admins easier. That’s why we’re excited to unveil flexible user roles.

Gain Granular Control

Now, you can control granular capabilities for your HackerRank users and fine-tune access across 30 different entitlements. 

These entitlements define whether users can take specific actions, such as editing tests, creating interviews, sending test invites, and more. 

Adjust Access at the Right Level

Need to adjust settings for everyone in a certain role? Just adjust capabilities at the user role level. For example, if you toggle off “Edit Tests” at the developer role level, you can disable your developers’ ability to edit tests.

You can also adjust entitlements at the individual level. If you don’t want all of your developers to have the ability to edit tests, but you want a specific user to have that ability, you can toggle on the ability to edit tests for that individual user.

Opt in to Flexible User Roles Today

Want this granular control for yourself, simply contact our support team at support@hackerrank.com or your dedicated account manager to opt in. For more detailed information, refer to our comprehensive support article.

The post Introducing Flexible User Roles: Right Size the Access for Your Users appeared first on HackerRank Blog.

The new AI-powered solutions, SkillUp and Engage, help organizations mobilize developer talent and accelerate tech hiring.

HackerRank is excited to share the launch of two groundbreaking solutions, SkillUp and Engage. These new additions to our Developer Skills Platform deliver more impact to help our customers attract, hire, upskill and mobilize their technical talent.

SkillUp analyzes existing developer skill proficiency, compares the results with target role proficiency, and then creates a personalized learning plan to build skills. Engage leverages AI to assist companies in creating and conducting real-world hackathons that are specific to their industry to attract and nurture top developer talent. The two solutions leverage the latest advancements in AI and help companies become GenAI ready.

We unveiled these new solutions during HackerRank’s AI Day Virtual Summit. The event featured keynote speakers including:

• Josh Bersin, HR tech visionary 
• Vinod Khosla, Co-founder of Sun Microsystems and the founder of Khosla Ventures
• Beyang Liu, Sourcegraph Co-founder & CTO
• Vivek Ravisankar, HackerRank Co-founder & CEO

“AI is changing software development rapidly,” said Vivek Ravisankar, Co-founder and CEO of HackerRank, on the impact of SkillUp and Engage. “Copilots are now an integral part of how developers build software. This means it’s going to change the entire developer lifecycle – the way you learn to code, how you get hired & how you upskill. Our two new products are helping companies attract & upskill developers in an AI-first world.”

“Every company we talk with is focused on building AI skills, but the domain is moving very quickly,” said Josh Bersin, Global Industry Analyst. “As the company who pioneered skills assessment for hiring, I am very excited to see HackerRank launch a solution for AI skills development. I believe SkillUp and Engage could transform this market and help companies truly credential their AI skills at this critical moment in the technology industry.

“As a customer of HackerRank, CodePath is excited about the launch of SkillUp and Engage as part of AI Day,” said Tim Lee, Chief Learning Officer and Founder of CodePath. “Through our partnership with HackerRank, we have had the opportunity to train the skills of thousands of developers and validate the skills that are being truly learned by using HackerRank’s assessments. The launch of SkillUp and Engage allows organizations like CodePath to utilize HackerRank badging and certifications without having to build and maintain their own system.”

Introducing SkillUp: Building Developer Skills for Productivity & Growth

SkillUp helps companies mobilize their internal talent to become GenAI ready. SkillUp analyzes existing developer skill proficiency and compares that with target role proficiency that informs a personalized learning plan to build skills for current and future success.  With the support of an AI Tutor, SkillUp accelerates hands-on learning that helps developers grasp new skills and technologies and apply those skills to solving real-world problems. 

Organizations can now have a strong understanding of their developers’ skills and help devise strategies to close the skills gap. The product is designed to be developer-first to empower them to be the best versions of themselves. 

Introducing Engage: Showcasing Your Tech Brand to Accelerate Hiring

Developers want to know what kind of challenges a company is working on. Job descriptions do a poor job of it. 

Engage leverages AI to assist companies in creating and conducting real-world hackathons that are specific to their industry/company to attract and nurture top developer talent. By entering their event goals, themes, target audience, and brand tone, AI creates all the elements required for a successful hiring event. With a library of real-world challenges, developers who participate experience, first-hand, the type of challenges they would face every day at the hiring company. 

This innovative solution not only showcases a company’s tech talent brand but also builds meaningful candidate relationships that can accelerate hiring now and in the future.

Who Are We? 

HackerRank, the Developer Skills Company, pioneered and leads the market for developer skills with over 2500 customers and a community of over 24 million developers. Companies trust HackerRank to help them set up a skills strategy, showcase their tech brand to developers, implement a skills based hiring process and ultimately upskill and certify employee skills…all driven by AI.

The post HackerRank Launches Two New Solutions to Mobilize Developer Talent and Accelerate Tech Hiring appeared first on HackerRank Blog.

What’s going on?

Since ChatGPT came onto the scene in late 2022, test after test has proven vulnerable to the wiles of generative AI. The initial GPT-3.5 model was impressive enough, and the more advanced GPT-4 has shown an even greater proficiency for test-taking. Name a large, well-known test, and ChatGPT has probably passed it. In addition to bar exams, SATs, and AP exams, ChatGPT has also passed 9 out of 12 AWS certification exams and Google’s L3 engineer coding interview. 

At HackerRank, we’ve seen firsthand how AI can bypass MOSS code similarity, the industry standard for coding plagiarism detection. 

All of these sudden vulnerabilities can seem scary for those administering tests. How can you trust the answers you’re getting? If your tests rely heavily on multiple choice questions, which are uniquely vulnerable to large language models, how can you revise test content to be more AI resistant?

These developments are worrying for test-takers, as well. If you’re taking a test in good faith, how can you be sure you’re getting a fair shake? Interviewing is stressful enough without having to wonder if other candidates are seeking an AI-powered advantage. Developers deserve the peace of mind that they’re getting a fair shot to showcase their skills. 

What’s our stance?

At HackerRank, we’ve done extensive testing to understand how AI can disrupt assessments, and we’ve found that AI’s performance is intrinsically linked with question complexity. It handles simple questions easily and efficiently, finds questions of medium difficulty challenging, and struggles with complex problems. This pattern parallels most candidates’ performance. 

However, creating increasingly intricate questions to outwit AI isn’t a sustainable solution. Sure, it’s appealing at first, but it’s counterproductive for a few reasons. 

• First, this could potentially compromise the core value of online assessments, weakening the quality of talent evaluation. More complex questions don’t automatically translate into better signals into a candidate’s skills. They take longer to answer, which translates into either longer assessments, or fewer questions (and fewer signals to evaluate). 
• Second, it would certainly degrade the candidate experience by focusing on frustrating AI rather than on giving developers a chance to showcase their skills. Losing sight of the developer experience tends to diminish that experience, which could result in more candidates dropping out of the pipeline. 
• Third, it would set up a game of perpetual leapfrog as more advanced AI models solve more complex problems, and even more complex problems are created to trip up more advanced AI. 

Instead, our focus remains on upholding the integrity of the assessment process, and thereby ensuring that every candidate’s skills are evaluated fairly and reliably. 

Introducing our new AI solvability indicator

Upholding integrity means being realistic—and transparent. This means acknowledging that there are assessment questions that AI can solve. And it means alerting you when that is the case, so you can make informed decisions about the content of your assessments. 

That is why we are introducing an AI solvability indicator. 

This indicator operates on a combination of two criteria. 

1. Whether or not a question can be fully solved by AI.
2. Whether or not that solution is picked up by our AI-powered plagiarism detection. 

If a question is not solvable by AI, it does not get flagged. Likewise, if a question is solvable, but the answer triggers our plagiarism detection model, it does not get flagged. The question may be solvable, but plagiarism detection ensures that the integrity of the assessment is protected. 

If a question is solvable by AI and the solution evades plagiarism detection, it will get flagged as AI Solvable: Yes. Generally, these questions are simple enough that the answers don’t generate enough signals for plagiarism detection to be fully effective. 

Questions flagged as AI solvable will be removed from certified assessments, but may still appear in custom assessments, particularly if those assessments have not been updated in some time. 

If you’re browsing through questions, you can also select to hide all AI-solvable questions, just as you can hide all leaked questions. 

What else is HackerRank doing?

Beyond the transparency of the AI solvability indicator, we are building in measures to actively ensure assessment integrity. These include: 

• AI-powered plagiarism detection. Our industry-first, state-of-the-art plagiarism detection system analyzes dozens of signals to detect certain out-of-bounds behavior. With an incredible 93% accuracy rate, our system repeatedly detects ChatGPT-generated solutions, even when they’re typed in by hand, and even when they easily bypass standard detection methods. 
• Certified assessments. Let us handle assessment maintenance. Our certified assessments are out-of-the-box tests curated and maintained by HackerRank experts. We take on all the upkeep, including keeping content current and flagging and replacing any leaked or AI-solvable questions. 
• Expanded question types. We’re expanding question types with formats and structures that are more resistant to AI solutions, such as projects and code repositories. These have the added benefit of being extremely close to the real-world environments and challenges your candidates would face in their daily work, giving you a true-to-life evaluation of their skills. 

What can you do?

No matter where your company stands on AI, we believe it’s best to be transparent about its capabilities. Yes, AI can solve simpler technical assessment questions. We prefer you to know that so that you can take informed actions. 

So what can you do? Every company is coming at AI in their own way, so there’s no one right answer. What works for one organization may not work for another. But broadly speaking, here are some steps you should consider to protect the integrity of your assessments.

• Stay informed. Yes, some technical questions can be solved by AI. At HackerRank, we help ensure assessment integrity through our market leading plagiarism detection and through solvability indicators that give you the transparency you need to deliver fair assessments. 
• Replace solvable questions. When a question in one of your assessments is flagged as AI solvable, a simple course of action is to replace it with an unsolved question from our library. We also recommend looking at the type of question you’re asking, and what you’re hoping to learn from it. It may make sense to replace a solvable question with an entirely different question type.
• Embrace new question types. Newer question formats like projects and code repos are more resistant to AI, and their close resemblance to real-world scenarios gives you a truer-to-life evaluation of how a candidate would perform in their daily work. 
• Take advantage of certified assessments. Don’t want to deal with maintaining and updating assessments? Let us do it for you. With certified assessments, HackerRank experts handle all of the content curation and monitoring, including replacing any leaked or AI solvable questions.
• Leverage HackerRank professional services. Have special needs for your assessments? Engage our experts for monitoring and content creation customized to your specific business objectives. 

Ensure assessment fairness and your own peace of mind

Ensuring assessment integrity in a time of rapidly advancing AI can seem difficult. You can only dial up question complexity so far before it starts to degrade the assessment experience and even compromise the value of assessments in finding qualified talent. That’s why we’re focused on reinforcing key pillars of assessment integrity, including our industry-leading AI-powered plagiarism detection, certified assessments, and solvability indicators that give you the transparency and signals you need to make the best decisions about your assessments. 

Be sure to check out our plagiarism detection page to go into more detail about how HackerRank is ensuring assessment integrity. 

The post AI Can Pass (Some) Test Questions. Now What? appeared first on HackerRank Blog.

25% of technical assessments show signs of plagiarism. 

While it’s impossible for companies to fully prevent plagiarism—at least without massively degrading the candidate experience—plagiarism detection is critical to ensuring assessment integrity. It’s important that developers have a fair shot at showcasing their skills, and that hiring teams have confidence in the test results. 

And the standard plagiarism detection method used by, well, everyone, is MOSS code similarity.

MOSS Code Similarity

MOSS (Measure of Software Similarity) is a coding plagiarism detection system developed at Stanford University in the mid-1990s. It operates by analyzing the structural pattern of the code to identify similarity, even when identifiers or comments have been changed, or lines of code rearranged. MOSS is incredibly effective at finding similarities, not just direct matches, and that effectiveness has made it the de facto standard for plagiarism detection. 

That doesn’t mean MOSS is flawless, however. Finding similarity doesn’t necessarily translate to finding plagiarism, and MOSS has a reputation for throwing out false positives, particularly when faced with simpler coding challenges. In our own internal research, we’ve found false positive rates as high as 70%.

AI changes the game

While not perfect, MOSS has been a “good enough” standard for years. Until the advent of generative AI tools like ChatGPT. 

ChatGPT has proven effective at solving easy to medium difficult assessment questions. And with just a bit of prodding, it’s also effective at evading MOSS code similarity. Let’s see it in action:

Step 1: We asked ChatGPT to answer a question and it did so, returning a solution as well as a brief explanation of the rationale. 

Step 2: Next, we directly asked ChatGPT to help escape MOSS code similarity check, and it refused.

Step 3: However, with some creative prompting, ChatGPT will offer unique approaches. And the way that ChatGPT’s transformer-based model works, it generates distinct answers every time, giving it a huge advantage in bypassing code similarity detection. 

Here are three different prompts and three totally different approaches. Note that ChatGPT transforms many variable names from the initial solution to evade code similarity checks.

Step 4: The moment of truth! When we submitted the revised answer through plagiarism detection, it passed cleanly. 

What’s the implication? 

Basically, MOSS code similarity checks can be easily bypassed with ChatGPT. 

Time to panic?

If MOSS code similarity can be bypassed, does that mean that technical assessments can no longer be trusted?

It depends. 

On one hand, it’s easier for candidates to bypass the standard plagiarism check that the entire industry has relied upon. So, yes, there is a risk to assessment integrity.

On the other hand, plagiarism detection has always been a compromise between effectiveness and candidate experience. MOSS is not intrusive, but its high false positive rates render it less definitive than it could be. Ultimately, it’s not really detecting plagiarism. It’s detecting patterns in the code that could be plagiarism.

Move over, MOSS

What happens now?

Plagiarism detection gets rethought for the AI era. Expect companies to scramble for better versions of MOSS, more complex questions, different question types, and more to make up the difference. 

At HackerRank, we’ve taken a different approach. While we’re always improving our question library and assessment experience, we’ve completely rethought plagiarism detection. Rather than relying on any single point of analysis like MOSS Code Similarity, we built an AI model that looks at dozens of signals, including aspects of the candidate’s coding behavior. 

Our advanced new AI-powered plagiarism detection system boasts a massive reduction in false positives, and a 93% accuracy rate. In real-world conditions, our system repeatedly detects ChatGPT-generated solutions, even when those results are typed in manually, and even when they easily pass MOSS Code Similarity. 

What happens when the example shown above gets submitted through our new system? It gets flagged for suspicious activity. 

Clicking into that suspicious activity reveals that our model identified the plagiarism due to coding behaviors.

What’s more, hiring managers can replay the answer keystroke by keystroke to confirm the suspicious activity. 

There’s nothing even close to it on the market, and what’s more, it’s a learning model, which means it will only get more accurate over time.

Want to learn more about plagiarism detection in the AI era, MOSS Code Similarity vulnerability, and how you can ensure assessment integrity? Let’s chat!

The post ChatGPT Easily Fools Traditional Plagiarism Detection appeared first on HackerRank Blog.

What you need to know

HackerRank has just launched its advanced plagiarism detection system, powered by AI. Designed to protect assessment integrity while ensuring developers have a fair and level playing field to showcase their skills, this system uses dozens of signals to detect suspicious behavior, including the use of external tools. 

The revolution will be prompted

AI is here, and it’s not going anywhere. 82% of developers have already experimented with some type of AI tool, and 55% are using AI assistants at work. AI is redefining what it means to be a developer, and in the future most code will be written with some kind of AI support.

AI opens up all kinds of exciting possibilities, but it can also muddy the waters of technical assessments. GPT-4 can not only pass AP exams and simpler coding challenges; it can also bypass MOSS code similarity, which has long been the industry standard for coding plagiarism detection. 

How can you ensure assessment integrity? If you’re relying on MOSS code similarity, you can’t. Not anymore. That means you can’t place full confidence in a candidate’s skills, and you can’t assure developers that they’re showcasing their skills on a level playing field. 

Fighting magic with magic

Basing any kind of detection system around AI usage would be a futile endeavor. AI is advancing so rapidly that keeping said detection systems ahead of it would be nearly impossible. 

Rather than relying on any single point of analysis like MOSS Code Similarity or AI usage, we took a different path and built an AI model that looks at dozens of signals, including aspects of the candidate’s coding behavior. This “defense in breadth” compensates for signals that may be bypassed on an individual basis and gives our new plagiarism detection system a more holistic way to detect shenanigans.

Think of it like a security system. A system that relies on a single factor, like a fingerprint scanner, is secure until that single factor can be bypassed. A multi-factor system that employs x-ray scanners, metal detectors, facial recognition, and gait analysis is far more challenging to sneak past.

Plagiarism detection, powered by AI

Our plagiarism detection system tracks dozens of signals across three categories—coding behavior features, attempt submission features, and question features—and analyzes them to calculate the likelihood of suspicious activity. 

After all of that crunching, our model predicts plagiarism suspicion as either High, Medium, or No. 

Importantly, we do not use any personal data such as gender, race, age, school, location, or experience in our analysis model. 

Currently, our advanced plagiarism detection system achieves an incredible 93% accuracy rate. In real-world conditions, our system repeatedly detects ChatGPT-generated solutions, even when those results are typed in manually, and even when they easily pass MOSS Code Similarity. 

What happens when a ChatGPT-generated answer gets submitted through our new system? It gets flagged for suspicious activity. 

Clicking into that suspicious activity reveals that our model identified the plagiarism due to coding behaviors.

What’s more, hiring managers can replay the answer keystroke by keystroke to confirm the suspicious activity. 

There’s nothing even close to it on the market, and what’s more, it’s a learning model, which means it will only get more accurate over time.

Ensure assessment integrity and candidate confidence

Our new AI-powered plagiarism detection is a groundbreaking innovation. It protects assessment integrity and ensures a fair playing field for developers to showcase their skills, and there’s nothing else out there that comes close.

As the AI revolution reshapes the industry, it’s vital to have reliable and efficient methods to detect and prevent plagiarism in online assessments. By analyzing various aspects of coding behavior and offering 93% detection accuracy, this system sets a new standard for maintaining transparency, fairness, and equity.

Want to go deeper and see our new plagiarism detection in action? We’d be happy to show you around.

The post HackerRank’s AI-Powered Plagiarism Detection Ensures Assessment Integrity in the ChatGPT Era appeared first on HackerRank Blog.

The post How Plagiarism Detection Works at HackerRank appeared first on HackerRank Blog.

Since its public debut in November, ChatGPT has taken the world by storm. In only five days, it surged to one million users. In just over a month, the valuation of the company behind it, OpenAI, grew to $29 billion.

Across sectors, there’s a growing chorus of questions about the implications of large language models (LLMs) like ChatGPT. Will these AI-enabled tools change education and make essay writing obsolete? Can they generate creative enough ideas to power mainstream ad campaigns? Will tools like ChatGPT provide a viable alternative to traditional search engines?

We’re asking some equally big questions ourselves: How well can ChatGPT actually code? And what impact will LLMs have on the broader world of computer programming? 

AI-powered innovation like ChatGPT is poised to fundamentally change the relationship between developers and coding, including how employers assess technical skills and hire developers. With that in mind, we dove deep into the details of ChatGPT, its impact on skill assessments, and what its development means for the future of technical hiring.

Key Takeaways:

• The coding potential of LLMs has reinforced the need for strategies and tools for upholding the integrity of coding assessments.
• Strong proctoring tools and plagiarism detection systems have become essential, and can help protect even solvable questions. 
• Employers should avoid multiple choice questions and problems that have answers so short that a plagiarism detection system can’t detect when a candidate has received help from a tool like ChatGPT.
• Continued growth of artificial intelligence will redefine the real-world application of coding skills and, in the process, change technical hiring as we know it.
• HackerRank is embracing AI and will pursue innovative ideas that imagine a future of programming in an AI-driven world.

What is ChatGPT?

On a basic level, ChatGPT is an example of a large language model. A large language model is a computer system trained on huge data sets and built with a high number of parameters. This extends the system’s text capabilities beyond traditional AI and enables it to respond to prompts with minimal or no training data.

The goal of ChatGPT’s developer, OpenAI, was to create a machine learning system which can carry a natural conversation. In practice, ChatGPT functions like a search engine or content creation system, synthesizing billions of data points into custom responses. 

Developing a Smart Conversational Agent

The development of ChatGPT incorporated two innovative approaches: 

1. ChatGPT is powered by the well-known ML model GPT3.5. The model is trained to complete the next few words of an incomplete sentence. The main idea behind this model is that, after training against billions of data points, the model starts to understand enough about the human world to complete sentences.
2. ChatGPT uses a human-in-the-loop system to continuously improve and answer questions in a more human-like fashion. OpenAI hired thousands of contractors to write human-like responses to challenging prompts as a way to continuously improve the model. Training the model to answer difficult questions improved ChatGPT’s responses at a remarkable rate.

Now that the training process is complete, users can run ChatGPT on accessible devices. This trait makes it superior to other models like AlphaCode, which are thought to be prohibitively expensive to run even after training is complete.

What Are the Strengths of ChatGPT?

Using the process above, OpenAI trained ChatGPT on almost all human knowledge. This enables ChatGPT to:

• Create never before seen sentences and code. Because it’s seen billions of sentences and lines of code, ChatGPT can synthesize the information it has seen and form answers to questions that can be perceived as novel. However, there’s no guarantee that this code will be correct or optimal.

• Combine ideas that it has seen separately but never in combination. For example, ChatGPT can write an answer to a coding question in the writing style of a specific author. 
• Exhibit a breadth of information. ChatGPT is trained on so much data that it has seen examples of most common situations and their potential variations. This enables it to give specific answers to niche questions or generalized answers based on more specific data.

What Are the Limitations of ChatGPT?

While ChatGPT outputs human-like sentences, and it’s easy to mistake its output as being powered by true intelligence, ChatGPT does have shortcomings. 

In describing the tool’s limitations, OpenAI explained that ChatGPT may occasionally “generate incorrect information” or “produce harmful instructions or biased content.” Industry publications have described ChatGPT as confidently wrong, exhibiting a tone of confidence in its answers, regardless of whether those answers are accurate. 

ChatGPT lacks the ability to fact-check itself or conduct logical reasoning. It often incorrectly answers questions and can be tricked relatively easily. Technologists have also noted its propensity to “hallucinate,” a term used to describe when an AI gives a confident response that is not justified by training data.

How ChatGPT Impacts Assessment Content

As a coding tool, ChatGPT excels at certain types of technical problems—but also has its limitations. A strong content strategy will be necessary to test your current coding challenges and prioritize the questions, and question types, that are less susceptible to AI coding support. 

ChatGPT has probably seen almost all known algorithms. But ChatGPT isn’t just able to answer these algorithm questions correctly. It’s also able to write new implementations of those algorithms, answer freeform questions, and explain its work.

As a result, ChatGPT can answer the following question types with reasonable accuracy:

• Well-known algorithms: It’s safe to assume that ChatGPT has seen and is able to answer all publicly available coding problems on platforms such as LeetCode and StackOverflow. If the algorithm appears in online forums or practice websites, ChatGPT will likely answer it correctly.

• Minor variations of problems. ChatGPT does well on variations that tend to add to the solution rather than change it in any substantial way. The system can, for example, easily reverse the order of an array of numbers.
• Multiple choice questions. When presented with a question and multiple potential answers, ChatGPT can usually identify the correct answer.

For hiring teams who administer coding challenges, that doesn’t mean you should necessarily avoid all questions that ChatGPT can solve. With the right protections in place, even questions solvable by AI can still be reliable. The key is to avoid questions that have answers so short that a plagiarism detection system can’t detect when a candidate has used a tool like ChatGPT. Even so, we are evolving our library with new types of content specifically designed with AI code assistance tools in mind.

Taking all of this into account, there are some actions you can take today to limit your hiring content’s exposure to the risk of plagiarism, including: 

• Avoid easily solved multiple choice questions
• Avoid simple prompts to solve for common or widely available algorithm variants
• Remove questions that require only a few lines of code to solve
• Use proctoring tools and plagiarism detection systems
• Combine coding tests with virtual interviewing tools to add empirical data to the hiring process

Ensuring Assessment and Hiring Integrity

In a world where humans and machines alike can write code, the ability to detect the use of AI-coding tools is invaluable. As such, employers increasingly turn to strategies and technologies that enable them to uphold the integrity of their technical assessments.

Assessment integrity has two core pillars: proctoring tools and plagiarism detection.

Proctoring Tools

One important component of ensuring assessment integrity is to build systems that provide the right proctoring capabilities. 

Proctoring is the process of capturing behavioral signals from a coding test, and its purpose is twofold. First, proctoring tools record data points that support plagiarism detection. Second, proctoring tools also act as a deterrent against plagiarism, as candidates who know that proctoring is in place are less likely to engage in such activity.

The key behavioral signals that proctoring tools often record include:

• Tab proctoring. Monitors if the candidate switches between tabs.
• Copy-paste tracking. Tracks if a candidate pastes copied code in the assessment.
• Image proctoring. Captures and records periodic snapshots of the candidate.
• Image Analysis. Analyzes webcam photos for suspicious activity.

Plagiarism Detection

In addition to proctoring tools, the integrity of an assessment also relies on plagiarism detection. In other words, the ability to flag when a candidate likely received outside help. 

The current industry standard for plagiarism detection relies heavily on MOSS code similarity. Not only can this approach often lead to higher false positives rates, but it also unreliably detects plagiarism originating from conversational agents like ChatGPT. That’s because ChatGPT can produce somewhat original code, which can circumvent similarity tests.

While the launch of ChatGPT caught many by surprise, the rise of LLMs has been a popular topic in technical communities for some time. Anticipating the need for new tools to ensure assessment integrity, HackerRank developed a state-of-the-art plagiarism detection system that combines proctoring signals and code analysis.

Using machine learning to characterize certain coding patterns, our algorithm checks for plagiarism based on a number of signals. Our model also uses self-learning to analyze past data points and continuously improve its confidence levels.

The result is a brand new ML-based detection system that is three times more accurate at detecting plagiarism than traditional code similarity approaches—and can detect the use of external tools such as ChatGPT.

Embracing Artificial Intelligence

As exciting as the launch of ChatGPT has been, LLMs with its capabilities are only the beginning. While it’s hard to predict the future, one thing is certain: AI technology is in a nascent state and will continue to grow at a rapid rate.

In the short term, the key to evolving your hiring strategy hinges on a renewed focus on content innovation and assessment integrity. By combining a strong question strategy with advanced proctoring and plagiarism detection, hiring teams can protect their assessment integrity and hire great candidates.

In the long term, we anticipate that artificial intelligence will redefine developer skills and, in the process, change technical hiring as we know it. 

At HackerRank, our mission is to accelerate the world’s innovation. As such, we welcome this new wave of technological transformation and will pursue innovative ideas that imagine a future of programming in an AI-driven world. 

Frequently Asked Questions

Can Your Plagiarism Detection System Detect Code From ChatGPT?

Yes. Our AI-enabled plagiarism detection system feeds several proctoring and user-generated signals into an advanced machine-learning algorithm to flag suspicious behavior during an assessment. By understanding code iterations made by the candidate, the model can detect if they had external help, including from ChatGPT.

When Will the Plagiarism Detection System Be Available?

The new plagiarism system is currently in limited availability, with plans for general availability in early 2023. If you would like to participate in our limited availability release, please let your HackerRank customer success manager know and we would be happy to enable you.

Can You Validate if My Coding Questions Are Easily Solved by ChatGPT and Provide Replacement Options?

If you would like assistance in verifying how ChatGPT responds to your custom coding questions, we can run a report and provide content recommendations based on the results. Please contact our HackerRank Support Team, who would be happy to help. 

Should I Avoid All Questions That ChatGPT Can Solve? 

No. HackerRank’s proctoring tools and plagiarism detection system can protect even solvable questions. Instead, avoid multiple choice questions and problems with very easy or short answers.

I Still Have Questions About ChatGPT. Who Should I Contact?

If you’re a customer looking for support on plagiarism and its impact on your business, you can contact your customer success manager or our team at support@hackerrank.com.

The post What Is ChatGPT? And What Does It Mean for Technical Hiring? appeared first on HackerRank Blog.

DMCA Takedowns. What are they? Why is HackerRank emailing me about one? And do I really have to remember another acronym? 

DMCA is a controversial topic in the developer community, as it can conflict with the free and open sharing of code that is so fundamental to the work we do as developers. As important as these values are, we here at HackerRank also believe that a soundly developed DMCA policy for our coding questions is vital. It’s the best way to ensure that all developers have an equal shot at landing job opportunities that match their unique skill sets and professional aspirations.

In this article, we break down the basics of DMCA, HackerRank’s DMCA policy, and how the whole process works.

What’s a DMCA Takedown?

The Digital Millennium Copyright Act (DMCA) is a United States copyright law that provides a legal framework for how copyright owners, online service providers, and users engage with copyrighted content. DMCA is a complex piece of legislation, but fundamentally, it established two key components:

1. It established “safe-harbor protections for online service providers if their users engage in copyright infringement.” 
2. It created a “notice-and-takedown system” which gives the copyright owner the ability to inform online service providers about a potential violation and request that they remove content from their site if it was used without permission. This is known as a DMCA takedown.

Failure to comply with the takedown notice waives the service provider’s right to safe harbor protection from infringement liability. This means the copyright holder can hold the service provider legally accountable.

HackerRank’s DMCA Principles

HackerRank’s mission is to accelerate the world’s innovation by focusing hiring decisions on skill, not pedigree. We do so by giving all developers the opportunity to showcase their skills in a fair and equitable testing environment. Each year, millions of developers solve problems during a HackerRank screening assessment or coding interview. The integrity of the questions that comprise these tests is critical for developers and employers to feel confident in their fairness and efficacy. 

DMCA isn’t a perfect system, and we recognize there are some drawbacks to pursuing a takedown policy. However, we’ve found that a proactive DMCA policy is the best approach for combating plagiarism, ensuring the efficacy of our tests, and providing a fair way for all developers to demonstrate their skills.

Accordingly, our larger DMCA approach centers on: 

• Ensuring a fair hiring opportunity for every developer by reducing plagiarism and upholding question integrity.

• Conducting an intensive manual review process to validate claims, with particular care taken to protect open source and developer communities from mistaken requests. 

Question leaks compromise the integrity of the test questions, making it difficult for employers to identify which developers have the strongest technical skills. To ensure that all developers get a fair shot at job opportunities, HackerRank has to uphold our question integrity by monitoring and containing leaked questions through DMCA takedowns.

However, we only send a DMCA takedown notice once we’ve conducted a diligent validation process and considered how the decision impacts developer communities. If the findings of our review are inconclusive, we err on the side of caution and refrain from issuing a notice.

HackerRank’s DMCA Policy

HackerRank has rebuilt its DMCA policy with a new prioritization process that ensures each takedown benefits the developer community and upholds assessment integrity while mitigating false positives. In the past, our policy was to take down any copied content that could be considered a violation of our copyright. We recognize that this was an incomplete approach, and our new DMCA policy combines automated detection with a thorough manual review process. First, our online detection system proactively identifies leaked questions on the internet. Then, we manually review every detection on a case-by-case basis. 

HackerRank’s DMCA takedowns follow a four-step process of identification, prioritization, manual review, and communication.

Identification:

• HackerRank actively monitors for leaked content by using automation to detect infringing material. 
• We then record the domain, question ID, and question creation date of each flagged URL.

Prioritization:

HackerRank prioritizes its DMCA responses based on the severity of the suspected breach and the type of domain, in addition to a range of other factors. We prioritize the takedown of pages that possess one or more of the following characteristics:

• Web pages that flagrantly post screenshots and/or verbatim copies of question text of regularly used questions.
• Web pages that, in reposting content, also infringe on the company’s logo and name.
• For-profit domains such as course or training websites that use HackerRank questions to drive transactions or user signups.
• Content that copies the default code stubs provided with each question. 

HackerRank deprioritizes takedowns of pages created by developer communities which publish content and developer-created solutions for learning and educational purposes.

Review:

• HackerRank’s product content team conducts manual, in-house reviews of every flagged URL to ensure that we make no false positives.
• Multiple reviewers assess each URL to ensure a fair and accurate review process.
• If we identify a false positive, we mark the question as not leaked and whitelist the URLs.
• If we determine that a case does violate HackerRank’s copyright and DMCA policy, we advance the content to the next stage. 

Communication:

• If a page warrants a takedown, HackerRank sends a DMCA notice to the online service provider.
• The notice will always contain the following:
  • Why the recipient is receiving the notice
  • How to respond to the notice
  • The required response timeline 
  • How to dispute a DMCA takedown

How to Respond to a HackerRank DMCA Takedown

When HackerRank sends a takedown notice, we will always include detailed instructions on how to respond to the notice. There are two potential courses of action:

Complying with the Takedown Notice

If HackerRank has directly contacted you with a takedown notice, we ask that you confirm in writing within fourteen days that you will comply with the request. If a domain has contacted you about a takedown notice from HackerRank, their policy will apply. You’ll then need to remove the content from your website. HackerRank will provide you with detailed documentation on the content and URLs we’d like you to remove.

Disputing a DMCA Takedown

We also recognize that DMCA is a complex issue and there may be cases where we’ve made a mistake. We encourage website owners that feel the takedown request is unfair or incorrect to dispute the DMCA notice. If you think we’ve made a mistake, you can reply back to the email or reach us at content-dmca@hackerrank.com. We’ll reevaluate the situation and retract our notice if we’ve made a mistake. 

Frequently Asked Questions

Are there any HackerRank questions that I can repost or share?

HackerRank has a community platform for developers to learn and practice coding. The thousands of questions there are publicly available and we encourage developers to repost these questions and share their solutions. You’ll never receive a DMCA notice for a question from our community platform. Each question also has a discussion forum where developers can discuss their approaches to the problem. 

Since I’m the creator of the solution code, shouldn’t I be able to share it?

If you’ve created an original solution to a HackerRank question, you’re free to share the code that you have written. However, we ask that you refrain from sharing our logo, the default code stubs, and the question itself.

I received a takedown notice from HackerRank and think it’s wrong. What should I do? 

You can dispute a DMCA takedown by replying to the email notice. Alternatively, you can reach us at content-dmca@hackerrank.com.

I’m a HackerRank customer and one of my questions leaked. Who should I contact?

If you’re a customer and you’re looking for support on DMCA, you can reach our Content Team at content-dmca@hackerrank.com.

I still have questions about DMCA. Where should I go?

We recommend the following resources for learning more about DMCA:

• The Digital Millennium Copyright Act 
• GitHub Guide to DMCA Takedowns
• The Importance of DMCA for Startups
• Unintended Consequences: Twelve Years Under the DMCA
• Tech Policy for Startups: Digital Millennium Copyright Act

The post How Does a HackerRank DMCA Takedown Work? appeared first on HackerRank Blog.

Imagine yourself as an ambitious developer. You’ve spent countless hours practicing your technical skills, and after weeks of applications, you finally hear back from one of your dream companies. They tell you that you’ll be taking a technical assessment, and while you’re a little nervous, you know you’re prepared to ace it. 

However, when test day finally comes, you don’t get a fair opportunity to show off your skills due to factors outside of your control. Maybe the assessment contained errors, or it didn’t keep candidates from copying answers from other test takers. No matter the reason, you feel cheated – and justifiably so. 

Unfortunately, many developers have dealt with poorly designed tests, and it’s prevented them from showing off their full potential. While hiring teams might be aware of this issue, they may not have the time, resources, or expertise to build and maintain job-related assessments that are fair and reliable. 

That’s why we’re proud to unveil Certified Assessments.

With Certified Assessments, the design, creation, and maintenance of tech screening tests are managed by HackerRank, so hiring teams can focus on the human side of hiring.

Leave Content Management to Us

Content management is a full-time job, and when hiring teams already have dozens of other responsibilities, it’s difficult to effectively manage test content. With Certified Assessments, we run 24/7 leaked question monitoring to find out when questions are leaked, so we can take immediate action to maintain the integrity of your tech screening tests. In addition, we carry out ongoing maintenance to ensure questions maintain their effectiveness and relevance. By delegating content management to us, hiring teams have more time to focus on other aspects of the hiring process, like building meaningful connections with candidates. 

“After creating each Certified Assessment, we conducted a formal content validation process.  This entailed gathering judgments from SMEs that each item requires one or more of the skills and abilities targeted by the test. But our work doesn’t stop with test creation. We conduct ongoing question monitoring to mitigate question leakage” stated Jeff Facteau.

Leverage the Strongest Plagiarism Detection on the Market

Every developer should have an equal opportunity to showcase their skills, so HackerRank runs advanced proctoring to mitigate malpractice and ensure no developer gains an unfair advantage. Each section of the test is unique and pulled from a bank of possible questions, so there is a 1 in 10,000 chance any two developers get the same test. We understand that all good developers use someone else’s code in their day-to-day, but when it comes to making a hiring decision, devs should let their unique skills speak for themselves.

Assess Job-Relevant Skills and Make Better Hires

We followed a rigorous job analysis and test development process for each of our
Certified Assessments. This ensures that they accurately assess the technical skills required for the targeted role (e.g, Software Engineer). Because the questions are designed to evaluate relevant job skills, and not irrelevant brain teasers, hiring teams gain a better indicator about the developer’s readiness to succeed in the role.

“We conducted a thorough job analysis for each role that included working with subject matter experts to identify and quantify the skills required for the roles. These findings were used to create a test plan for each Certified Assessment and identify the questions to include in them,” said Jeff Facteau, Chief I/O Psychologist at HackerRank.

Give Every Developer an Equal Opportunity to Showcase Their Skills

Bias can influence decision-making and prevent people from having a fair shot to show off their talents. Every HackerRank Certified Assessment undergoes a third-party fairness and sensitivity review to remove bias regarding race, gender, and individual disabilities. These reviews ensure every developer has an equal opportunity to showcase their skills and land their dream role.

“We had an independent consulting firm review our test items for sensitivity and bias. In addition, we fully documented the test creation and validation process in a technical report that conforms with the U.S.’ Uniform Guidelines on Employee Selection Procedures,” said Jeff Facteau.

We all have work to do before we can create a world where skills speak louder than words. At HackerRank, this is our mission, and Certified Assessments brings everyone closer to achieving it. For more information on Certified Assessments, check out our support article, or schedule a demo to see Certified Assessments in action.

The post Certified Assessments: Leveling the Playing Field for Every Developer appeared first on HackerRank Blog.