Finding a Way Through the Dark
No source code, dozens of applications. Keep the lights on.
They all just walked out
Names, technologies, and certain identifying details have been changed to protect the identities of the organizations and individuals involved. The underlying events are based on my own experiences.
At one point in my career, long ago, I worked with Company X. Before I arrived, a dispute had resulted in the departure of an entire engineering team. Somehow, the source code had disappeared with them. Legally, I’m not sure how that’s possible but I wasn’t around for those conversations.
I had to keep the lights on, but how?
There were dozens of web applications, spanning multiple generations of technology, all running across two servers. Several hundred databases, more stored procedures than you could shake a stick at, and scheduled ETL jobs to boot! No source code?! Really?!
Week-one tasks included finding and updating text in dynamically generated emails or changing menus in Department Y’s applications. This yielded some basic understanding of how Department Y had been limping along, technically speaking.
I invented a way to see in the dark
Department Y would ask for a data dump for Application Z. I used a popular decompiler program to inspect compiled binaries to find out which stored procedures the applications called. Then I’d check the stored procedures, which led me to the tables.
I did this a few times before I wondered, “Huh, how does this third-party decompiler do this?”
That search led to a lower-level disassembly tool. After learning its invocation flags, I wrapped it in a console app that could search all the compiled application binaries in nested folders for a given string.
Just like that, I had a way to see which applications invoked a specific stored procedure.
This saved me tons of time. A few days later, I added a feature to list the stored procedures a given application invoked. I was now able to answer Department Y’s questions much faster.
First make it possible, then make it good
I began decompiling the applications and restoring them to buildable projects. The recovered source code was gnarly from decompilation artifacts and lost metadata, but at least it could build. After a few rounds of testing, I was able to safely produce replacement programs to run in a staging environment and eventually production.
Over time, I began checking decompiled, runnable projects into a new code repository server.
The Great Server Migration
Among the servers that Department Z managed were the servers hosting all of Department Y’s applications and databases.
Department Z was migrating to a newer server version, and it was our job to make sure all the applications still ran smoothly. Only a handful of web applications had dependencies that broke on the new server version.
Around this time, one of my colleagues joined me to get them upgraded before the migration. On C-Day (Cutover Day), we ran our preflight checklist and made sure systems were ready.
The migration ran late into the evening but was ultimately successful.
The Email Queue
A few months later, Department Y reached out to me about a specific application’s form that was rejecting submissions.
This was one of the few web applications I hadn’t had the pleasure of decompiling yet. I found the error logs. “Error code 6”, which was somehow less descriptive than logging “walrus parts”.
Hmm. What does the submit button invoke? It just serializes the form and sends it into a stored procedure. The stored procedure writes to an email queue table.
I was well acquainted with the email queue table implemented by Department Y’s former engineering team. A scheduled job read from it once a minute and updated it after processing.
--illustrative code, not actual code.
SELECT QUEUE_ENTERED_DATE, MESSAGE_CONTENTS, PERSON_SENDING, PERSON_SENDING_TO FROM
xyz.SOME_EMAIL_TABLE
WHERE THE_DATE_EMAIL_WAS_SENT IS NULL;For each record, it sent an email.
Yep. The bodies were in plaintext in the database.
I was hoping to see a familiar redacted pattern for the sensitive information stored in the body.
Nope.
The information was sitting there in plaintext, unredacted, and being emailed to another department. I could almost hear the former engineering team’s hypothetical conversations about security and just getting it done.
I immediately notified Department Y. They instructed me to take the site offline and address the exposed data.