SentinalAI pulls 65 features out of a source file, lexical counts and Python AST structure, and tests whether that signature alone separates human-written code from AI-generated code. No semantic analysis, no execution, no model-in-the-loop judgment call. Just a fixed feature vector and a classifier trained on labeled pairs.
The hard part isnât extracting features. Itâs building a corpus where the label is actually clean. SemEval-2026 Task 13 gives scale: 500K samples spanning 34 AI generators across Python, C++, and Java, pulled from DaniilOr/SemEval-2026-Task13 on HuggingFace. But scale alone doesnât control for confound: a classifier trained on that corpus might just be learning âwhat kind of problem gets solved by which modelâ rather than âwhat AI-written code looks like.â
So the second dataset removes that confound by construction. 96 competitive programming problems from Codeforces, each one solved twice: once by a human, pulled from GitHub before 2022, and once by deepseek-coder:6.7b, run locally through Ollama. Same problem, same constraints, two authors. Any stylistic signal the classifier picks up here has to come from how the code was written, not what it was asked to solve.
Evaluation on the paired corpus uses leave-one-problem-out cross-validation, holding out every sample tied to a given Codeforces problem when testing on it. Thatâs a stricter bar than a random split, because it forces the classifier to generalize past problem-specific vocabulary and idiom rather than memorizing a particular solution pattern.
The two datasets donât agree on how AI-generated code is defined. One draws from 34 different generators and whatever distributional quirks each one has. The other draws from a single 6.7B model running locally. test_generalization.py and test_generalization_reverse.py exist specifically to probe that gap: train on one dataset, evaluate on the other, then flip the direction. A stylometric feature set that transfers across both is a much stronger claim than one that only works within a single corpusâs assumptions about what âAI-generatedâ means.
The same Ollama instance that generates the Codeforces AI solutions also runs the perplexity validation step in scripts/, giving a second, independent signal to check the stylometric features against on the same corpus.
Thereâs an inference stub in api/, currently just scaffolding. The research question comes first: does a 65-dimension lexical and structural fingerprint hold up across generators and across problem domains, or does it collapse the moment the training distribution shifts.