SentinalAI is an empirical study, not a product. The question itâs built to answer: do lexical and structural style features, extracted deterministically from source code, carry enough signal to reliably separate AI-generated code from human-written code. The repository is organized around that question end to end, from corpus collection through feature extraction to classifier evaluation.
Feature extractor
features/ implements a 65-feature extractor combining lexical counts with Python AST structure. This is the only representation the classifiers see. No LLM judges the code, no embedding model scores it, no execution trace is captured. Every feature is computed by deterministic code before a classifier ever touches the sample.
Two corpora, two failure modes
The first dataset, SemEval-2026 Task 13, is pulled directly from DaniilOr/SemEval-2026-Task13 on HuggingFace: 500K samples across 34 distinct AI generators, spanning Python, C++, and Java. Its value is coverage. A feature set that only works against one modelâs output style is not detecting âAI-generated code,â itâs detecting one modelâs habits. 34 generators forces the feature set to find whateverâs common across a wide range of generation styles, if anything is.
Its weakness is confound. At this scale, human and AI samples arenât guaranteed to be answering the same underlying problem in the same way, so a classifier can pick up signal that has nothing to do with authorship.
The second dataset exists to close that gap. collector/scrape_codeforces.py and collector/fetch_github_human_solutions.py build a paired corpus: 96 competitive programming problems from Codeforces, each solved by both a human (sourced from GitHub, pre-2022) and an AI model (deepseek-coder:6.7b, generated locally via Ollama). Because both solutions answer the identical problem statement, any stylistic difference the classifier finds is attributable to the author, not the task.
Pipeline
The scripts run in a fixed order. collector/pull_semeval_dataset.py pulls the SemEval corpus. scripts/build_feature_matrix.py runs the 65-feature extractor across it. models/train_baseline.py trains a classifier on that matrix. In parallel, scripts/build_paired_features.py builds the feature matrix for the Codeforces pairs, and scripts/train_paired_classifier.py trains against it using leave-one-problem-out cross-validation, holding out all samples for a given problem when evaluating on it. That protocol is deliberately stricter than a random train/test split: it prevents the classifier from just memorizing surface patterns tied to a specific problemâs vocabulary.
scripts/ also includes a perplexity validation step, scored through the same Ollama-hosted deepseek-coder:6.7b instance used for AI generation, giving an independent check against the stylometric features on the same samples.
Cross-dataset generalization
scripts/test_generalization.py and scripts/test_generalization_reverse.py train on one corpus and evaluate on the other, in both directions. This is the sharpest test the repo runs: SemEvalâs 34-generator diversity and the paired corpusâs single-model, same-problem design represent two different definitions of âAI-generated code.â A feature set that transfers across both directions is evidence the 65 features capture something about generation style itself, not an artifact of one datasetâs construction.
Setup
The pipeline depends on Ollama running locally at localhost:11434, with deepseek-coder:6.7b pulled, for both AI code generation and perplexity scoring. api/ contains a stub inference endpoint; it is not the focus of the current work, which is the corpus-and-classifier evaluation described above.