Radiant Digital Helped a Telecom Provider Classify Unstructured & Structured Data by Its Contents A Fortune 500 telecommunications provider was governing an analytics estate of hundreds of thousands of columns across thousands of schemas and more than 100 Google Cloud projects, classifying each column against approximately 250 governance policies using column names, manual review, and expert correction. Unstructured columns could not be inspected at all, so they were blanket restricted. 

Radiant Digital designed a tiered classification approach that examines actual data values and uses manual review when needed. In testing, it classified every column across the tested loads of 10 to 500 columns, averaged about 0.47 seconds per column, and reached 93% confirmed classification accuracy after reconciliation with governance analysts. Separately, unstructured data detection reached 80 to 82% average accuracy.  

Classification based on names left accuracy gaps at scale 

The telecom provider had manually certified application compliance with its security policies. As data was onboarded to Google Cloud for analytics, structured columns were classified from table and column definitions, with an AI model assigning a policy and people correcting it where needed. That approach was accurate about 75 to 85% of the time. 

  • Non-standard naming and missing descriptions caused misclassification.
  • Thousands of catalog columns relied on subject matter expert review.
  • Thousands of operational databases have not yet been analyzed.
  • The reference data used to measure accuracy had known quality gaps of its own.
  • Chat transcripts, JSON blobs, and free form notes had no way to be inspected, so they were blanket tagged at the highest restriction level and legitimate access was over restricted. 

Radiant Digital designed for content, cost, and human judgment 

Radiant Digital's thinking shaped four design choices. 

  • Profile the unstructured & structured data, not just name: For unstructured data, pattern matching catches well defined formats such as social security numbers, credit card numbers, and IP addresses; entity recognition handles context dependent fields, and a language model handles only ambiguous cases. For structured data, the classifier reads actual column values across six data types and maps each column against approximately 250 policies.
  • Run the simplest method first. Each column moves through pattern matching, keyword lookup, and semantic similarity before a language model is considered, and exits at the first step that meets the confidence threshold. This keeps cost down and makes the approach scalable to large datasets.
  • Use manual review when needed. The classifier returns a confidence score, detection method, reasoning, and review flag, so uncertain results can be examined. Audit logging records how each classification was reached.
  • Change policies through configuration, not code. New policies can be added without code changes, so the policy set can grow without new development. 

It fits inside the workflow the team already uses 

When the telecom provider’s existing onboarding process is uncertain about a structured column’s policy, Radiant Digital’s classifier examines the values in that column. Across the structured and unstructured data capabilities, Radiant brings together Aho Corasick pattern matching, GLiNER entity recognition, semantic search using sentence transformer embeddings, and a language model for ambiguous cases. Applying the appropriate methods in sequence helps resolve clear cases efficiently and reserves more intensive analysis for uncertain ones. 

The capabilities work within the provider’s Google Cloud environment. Both run as FastAPI services on Cloud Run, with BigQuery holding source data and structured classification results and audit logs. Cloud SQL for PostgreSQL with PGVector supports semantic search.  

Parallel processing and batched language model calls support efficient structured data classification. Radiant Digital also prepared Responsible AI documentation for the provider’s review. 

Radiant Digital built a solution where none existed and proved it with experts 

The telecom provider had no solution for its unstructured data. Radiant Digital worked independently, applied the provider's own algorithms, and used AI across the full software development lifecycle. With scarce test data and slow manual validation, the team built its own methods to check each mapping, then used multiple passes to improve accuracy and efficiency while cutting language model usage. 

Many mismatches against the reference data turned out to be more specific classifications, not errors, because the reference data had known quality issues. Reviewing results with the provider's governance analysts made the accuracy benchmark defensible. 

Results confirmed in load tests and expert review 

  • 93% confirmed classification accuracy after reconciliation with the provider's governance analysts. This is reported separately from the 80 to 82% average detection accuracy for unstructured data.
  • 100% classification success at every load tested volume, from 10 to 500 columns.
  • About 0.47 seconds average processing time per column.
  • More than 15,000 unique columns tested.
  • For unstructured data, detection reached 80 to 82% average accuracy, and access can now be classified column by column instead of through one blanket restriction. This capability is in production.
  • The structured data classifier is moving into production within the provider's existing workflow. 

Designed to keep learning and to scale 

Confirmed results feed back into the system every day, so accuracy keeps improving. The semantic search foundation built for unstructured data now also powers structured data classification; new policies can be added through configuration, and the design extends to future policies and scales to large datasets. The provider now has a governance foundation that can grow with its data, not a one-time fix. 

Ask Radiant Digital to test column classifications against actual data values and identify where classification based on names leads to incorrect policy assignments. 

Related Posts