Gmail-to-Azure Ingestion System

Customs Data Automation - Abu Dhabi

Objective

The client needed a way to stop manually sorting through hundreds of emails
coming from Abu Dhabi Customs. The goal was straightforward: build a system
that reads those emails automatically, pulls out the important data, saves
the attachments somewhere reliable, and gives the team a simple screen
to search and review everything.

Key Features and Functions

• Connects to Gmail through OAuth2 and pulls only emails sent from adcustoms.gov.ae everything else is ignored.


• Extracts structured fields from the email body: Request ID, Job Number, Importer, Consignee, Shipper, and Agent name, using pattern matching.


• Automatically uploads all attachments (PDFs, images, documents) to Azure Blob Storage, organized by year, month, and message ID.


• Stores everything in a MySQL database with full deduplication the same email is never processed twice.


• Runs on a schedule (every 5 minutes via cron), no manual triggering needed.


• Includes error handling at every step. If something fails, it logs the error and records it in the database instead of crashing silently.

User Experience

The admin dashboard is clean and dark-themed. When you log in, you see
summary cards at the top showing how many emails were processed today,
how many succeeded or failed, and the total attachment count. Below that
is a table you can filter by Request ID, Job Number, date range, or status.
Clicking any row opens a detail page with all the extracted data and
direct download links to the attachments on Azure. The login is
session-based with an option to restrict access by IP address.

Added Value

Before this system, someone on the team had to open each email, copy the
reference numbers into a spreadsheet, and manually download and rename
attachments. That process took hours every day and mistakes were common.
Now it happens in the background with no human input. The team just opens
the dashboard when they need to look something up. Processing time went
from hours to seconds, and nothing gets lost or duplicated anymore.