Applied machine learning · Dataset engineering
AI Code Testing
The project begins with the missing asset: a traceable multiclass corpus of Python defects. It acquires, labels, de-duplicates, filters and balances material from three source categories before evaluating a TF-IDF and XGBoost classifier across seven final classes.
- Python
- Corpus engineering
- TF-IDF + XGBoost
The problem
Static-analysis tools can report that a rule was broken, but that output is not automatically a useful multiclass dataset. This project asks a different question: can a Python snippet be classified as a syntax, runtime, logical, security, performance or compatibility issue, or as correct code?
The model is only the final stage. Before training was possible, the work needed a taxonomy, acquisition paths with different strengths and a traceable method for turning noisy candidates into labelled examples.
Evidence gallery
Corpus and reproduced classifier results at a glance. 
Three raw-source paths feed one curated seven-class classifier pipeline.
Why the data was the hard part
There was no ready labelled corpus that matched the seven-way question. Repository files supplied realistic code but overwhelmingly looked correct; Stack Overflow supplied small discussion-shaped blocks; synthetic generation could target missing defect types but introduced repeated structures. No one source could carry the task alone.
The engineering work was therefore a sequence of data decisions: retain source lineage, combine unlike inputs, remove exact duplicates, map analysis signals into a shared taxonomy, inspect classified blocks, filter unusable material and rebalance the result without pretending that every label was human ground truth.
That sequence is still countable. It starts with 7,749 acquired candidates, reaches 11,511 after the retained source pools are combined, falls to 7,646 unique items, and finishes at 3,502 labelled samples representing 3,495 unique code-and-label pairs. The final classifier is meaningful only because this chain can be explained.
Three sources, three strategies
Five vendored public Python repositories contributed 3,975 files through a local repository walk. The Stack Overflow API contributed 2,774 Python-tagged candidates using pagination, rate limiting and HTML code extraction. A generator supplied 1,000 additional examples from 31 hand-written defect templates.
Together these paths acquired 7,749 raw candidates. Repository code provided breadth, question-and-answer blocks provided varied short examples, and templates supplied controlled cases for categories that the first two sources did not cover evenly.
Evidence gallery
Scroll horizontally to inspect the chart
7,749 candidates acquired from repositories, Stack Overflow and controlled generation.
Labelling, and its honest limits
Labels were assigned programmatically: repository and Stack Overflow blocks were mapped from pylint and flake8 output, while generated samples inherited the class of their template. A manual correction pass then reviewed classified blocks.
This is not expert-annotated ground truth. Correct Code also acts as a fallback bucket when the mapped signals do not establish another class, making it broader and more lexically varied than a tightly defined defect label. That choice helps explain later confusion with Syntax Error rather than hiding it behind one headline score.
Curation changed the dataset
The retained source pools produced 11,511 candidates before exact de-duplication reduced them to 7,646 unique items. Later filtering and class balancing produced the final 3,502 samples. The largest-to-smallest class ratio moved from roughly 4.7:1 in the raw acquisition to 1.43:1 in the final corpus.
Curation also changed the taxonomy. Style Error travelled through much of the pipeline but was retired before final training, leaving seven classes. Removing a label whose evidence no longer justified it was a data-design decision, not a missing eighth output.
| Class | Samples | Share (%) |
|---|---|---|
| Logical Error | 590 | 16.85 |
| Syntax Error | 534 | 15.25 |
| Correct Code | 523 | 14.93 |
| Performance Issue | 505 | 14.42 |
| Runtime Error | 486 | 13.88 |
| Compatibility Issue | 451 | 12.88 |
| Security Issue | 413 | 11.79 |
| Total | 3502 | 100.00 |
Evidence gallery
The retained source pools move from 11,511 candidates to 7,646 unique items and 3,502 final samples. 
The final corpus contains seven classes with a 1.43:1 largest-to-smallest ratio.
Features and model
The curated text is transformed by TF-IDF into 6,457 features and classified with XGBoost. The final evaluation uses an 80/20 split with random_state=42: 2,801 training samples and 701 held-out samples.
This intentionally simple representation also sets the ceiling. The vectoriser's default token pattern drops punctuation and single-character tokens, even though characters such as colons and brackets can carry decisive syntactic meaning in Python.
Results, with the method attached
A clean reproduction with one consistent label encoder reached 81.6% accuracy, 0.820 macro F1 and 0.818 weighted F1 across seven classes on 701 held-out samples. Per-class F1 ranged from 0.67 for Syntax Error to 0.94 for Performance Issue.
The original notebook derived training and test label maps independently, so its stored evaluation could not be treated as the final result. Re-running the same pipeline with one encoder corroborated it to within one percentage point and made the class mapping reliable.
The confusion between Correct Code and Syntax Error accounts for 22% of all errors. That result closes the loop with the fallback label and the token representation: some syntactic defects lack the lexical cues the model sees. The charts are accompanied by semantic tables so every value remains available without relying on the image.
| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| Compatibility Issue | 0.9200 | 0.8023 | 0.8571 | 86 |
| Correct Code | 0.6609 | 0.6786 | 0.6696 | 112 |
| Logical Error | 0.8547 | 0.8929 | 0.8734 | 112 |
| Performance Issue | 0.9596 | 0.9223 | 0.9406 | 103 |
| Runtime Error | 0.8286 | 0.7909 | 0.8093 | 110 |
| Security Issue | 0.9186 | 0.9186 | 0.9186 | 86 |
| Syntax Error | 0.6346 | 0.7174 | 0.6735 | 92 |
| Macro average | 0.8253 | 0.8176 | 0.8203 | 701 |
| Weighted average | 0.8220 | 0.8160 | 0.8180 | 701 |
| Accuracy | 0.8160 | 701 |
| Actual / predicted | Compatibility Issue | Correct Code | Logical Error | Performance Issue | Runtime Error | Security Issue | Syntax Error |
|---|---|---|---|---|---|---|---|
| Compatibility Issue | 69 | 7 | 1 | 0 | 1 | 1 | 7 |
| Correct Code | 3 | 76 | 6 | 2 | 5 | 3 | 17 |
| Logical Error | 0 | 3 | 100 | 0 | 2 | 0 | 7 |
| Performance Issue | 0 | 5 | 0 | 95 | 1 | 0 | 2 |
| Runtime Error | 1 | 11 | 4 | 0 | 87 | 3 | 4 |
| Security Issue | 1 | 1 | 0 | 0 | 4 | 79 | 1 |
| Syntax Error | 1 | 12 | 6 | 2 | 5 | 0 | 66 |
Evidence gallery
Precision, recall and F1 for every class on the 701 held-out samples. 
The confusion matrix exposes where the seven-class representation succeeds and fails.
What four fresh examples reveal
A tiny qualitative check used four portfolio-authored snippets that were never executed and were not drawn from the corpus. The predictions agreed with the intended reading for two of the four: the security example and the well-formed helper. A missing colon and a runtime-risk example both fell back to Correct Code.
This is not a second accuracy measure. It is a compact generalisation probe that shows why a larger model is not the first improvement. The next step is a richer representation—parse success, AST shape and token-type n-grams—followed by measured label quality and group-aware splitting so template variants cannot span training and test sets.