Too Many Files? Feed Them to Python’s Voracious Glob Module

Clinton’s post on using Python to process multiple CSVs was a huge help in conducting analysis! The write up is easy to follow, perfect for the novice user like me to follow and references an earlier post that covers the basics.

Clinton Brownley's Decision Analytics

In my previous post I mentioned that a coworker had recently emailed me a folder full of over two hundred Excel files and asked me to extract some relevant data from each file. I noted how undertaking that task manually would have been time-consuming and error-prone and described how exhilarating it was to accomplish the task quickly by writing some Python code.

I didn’t show how to process multiple files with Python in that post because it is easier to understand the code for processing multiple files once you’re familiar with code for processing one file. So that’s why in that post I demonstrated how to read and write a single CSV file with Python. With that knowledge under our belts, we’re now prepared to understand Python code for processing multiple CSV files.

One good way to learn to code in Python is to create small datasets on your laptop…

View original post 1,808 more words


Drive and data recovery…

A friend of mine sent me an MMS the other day with a picture of the infamous Microsoft blue screen of death (BSOD) and blames his wife of a malware infection while surfing for Elmo videos for their 2 year old son.

Having established the premise for the post, my friend lent me his drive in my request to help him recover his personal files before he reloads his operating system of choice.   Assuming malware caused the BSOD, I would extract his user files onto another medium and return the corrupt drive and his files in a days time.  Done deal.  Easy.  Hell, I was going to acquire an image and see if I could find the malware for fun (because that is how we learn, no?)

This post will walk-through how I recovered my friend’s data.  If this post helps one person other than myself, it will have achieved the goal.  Otherwise, I’ll use this post to refresh my memory when I do this again.

Using a drive duplicator, with USB 3.0 output, I connected my friend’s SATA drive to a Linux box running a lubuntu distro and connected a spare external to another USB interface.  To copy the data over, I decided on using ‘rsync‘ from the command-line interface (CLI).
FriendDrive: /dev/sdd
FriendDrive volume of interest: /dev/sdd3 mounted as /media/jc0re/8A54DE6254DE5095
SpareDrive: /dev/sdc
SpareDrive volume for use: /dev/sdc2 mounted as /media/jc0re/JC0R3

^^ This stuff was all automatic upon plugging in the drives but listed for reference
Using rsync to copy the entire user profile, we use the following:

sudo rsync -va –progress /media/jc0re/8A54DE6254DE5095/Users/GoodFriend /media/jc0re/JC0R3/GoodFriend

After about 15 minutes, I start seeing “file has vanished: /location/of/file/being/copied/here
Looking back over the progress, (hence “–progress” parameter in the initial command), I see multiple “Input/output error (5)“. I also notice the volume is no longer mounted and the drive is no longer recognized. I repeat the rsync several times and the same results. Without too much more thought, I’m going to assume this drive is going bad and the I/O errors might be why Microsoft is throwing BSODs while watching Elmo.  It could also be a bad wire.

With the drive failing and disconnecting when trying other various means to copy the data to no avail, I move the focus onto recovering what I can with a tool called dd_rescue. I “sudo apt-get install gddrescue” to get ddrescue on my box for a raw image of the GoodFriend drive. After install, I follow the ForensicWiki ddrescue reference on trying to get the data.
Before proceeding, I ran into a format error and was limited to a 4096MB image file causing ddrescue to stop. Make sure the drive is formatted in something other than FAT because it will not support an image larger than 4GB. I used this link for refreshing myself on fdisk commands.

sudo ddrescue –no-split /dev/sdd3 /media/jc0re/JC0R3/GoodFriend logfile
^^ This simply tries to grab what it can from the drive. I was able to get 80MB of 500GB. Into the freezer it went.

sudo ddrescue –direct –max-retries=3 /dev/sdd3 /media/jc0re/JC0R3/GoodFriend logfile
^^ This is the first attempt to go after the broken stuff with uncached reads (we’ll cover uncached reads in a shorter post another day). I was able to get 300GB of 500GB. Back into the freezer. And change my cable.

sudo ddrescue –direct –retrim –max-retries=3 /dev/sdd3 /media/jc0r3/JC0R3/GoodFriend logfile
^^ This is the second and last attempt to go after the broken stuff but by the full sector. I was able to get 499GB of 500GB. Done. Finally.

Next Day: We have the ‘GoodFriend’ raw image on an external USB drive.  Using the linux box, you can mount the image file to a predetermined mount point and examine the contents of the image as if it’s a real, physical drive. Nifty. We no longer have to worry about the physical ailments of the failing hard drive.

To mount from the console, we use this link for reference:

sudo mount -o loop /media/jc0re/JC0R3/GoodFriend /mnt/extra
^^ For the Microsoft users: In this scenario, when you navigate to /mnt/extra, you are looking at the root of GoodFriend’s hard drive.  The ‘/mnt/extra‘ is simply a reference pointer.

To copy the actual user data, we need to extract them from the image file and place them onto another medium. I used the same hard drive dock from the previous post and slapped a recently wiped and formatted 500GB SATA into it.  Unfortunately, the drive was formatted with my Mac and using HFS+.  Linux didn’t play well and so we had to do some extra work.  To mount and use the drive on the linux box, we needed to use hfsprogs, a port of Apple’s tools for working with HFS+.

sudo apt-get install hfsprogs
^^ This installed the hfsprogs package

After install, we try to remount the HFS+ drive
sudo mount -t hfsplus -o remount,force,rw /dev/sde2 /media/jc0r3/Untitled
^^ This remounts the HFS+ drive (/dev/sde2) and gives root permission to the drive

Now, we try to rsync my GoodFriend’s user profile from the image file to the HFS+ drive using a similar command from the previous attempt in the earlier post
sudo rsync -va –progress /mnt/extra/Users/GoodFriend /media/jc0re/Untitled/

This leaves me with GoodFriend’s user files for which I’ll burn onto a DVD or on a USB drive for him to restore to a new system!

w32/PoisonIvy Analysis

Crafting a one-page executive summary on a particular string of malware for leadership or particular client can be challenging for an IT security professional.  Without getting into the nitty-gritty, how do you explain it?  I believe some of the key points may include some background info, functionality summary, effects on user’s privacy, and any details on attribution as everyone seems to be interested in who created it and their motive.  Enjoy a short read on PoisonIvy Remote Access Tool (PIVY RAT)

Considered a lynch pin in many attacks, PIVY’s GUI drives popularity with noobs and script kiddies.   The install vector for PIVY varies amongst the common methods for proliferating malware such as drive-by downloads and e-mail attachments.  As a remote access tool, PIVY doesn’t necessarily take advantage of any particular vulnerability as it is infamously delivered within the payload of an attack.  This tactic was evident in the RSA and Nitro campaign attacks where zero-day vulnerabilities were exploited and PIVY was deployed to open backdoors for the adversary.  Freely available, PIVY enables remote access, keylogging, screen and video capturing, file transfers, system administration, password theft and traffic relaying.[1]  PIVY breaches a user’s privacy by allowing an attacker to view the victim’s screen output, record the victim’s key strokes, and access the victim’s system without their permission.  Generally, malware is surreptitiously developed and publicly disclosed when security professionals analyze compromises that utilized the said malware.  The history of PIVY dates back to 2006 with v2.1.2[2] and from various, unverifiable discussion boards, a developer by the name of ‘Jonas’ was the author.  Predominantly associated to attacks utilizing PIVY as means to spread or persist, the PIVY origin was not identified.

Receiving the majority of attention, the 2011 RSA breach was supplemented through the use of a PIVY variant.  The attack vector was a targeted phishing email with an Excel attachment exploiting a zero-day Adobe Flash vulnerability (CVE -2011-0609)[3].  Careful lateral movements across the network to servers of interest and data extraction with FTP completed the attack in a nutshell.  The orthodox methodology used in the RSA attack could have been mitigated with spear phishing training, email content checking, outbound FTP filtering, internet access proxying, improved internal protections on servers e.g. dual-factor auth for admin access, internal network access controls and segregation, and lastly, better application management to prevent an .xls from executing Flash content.

[1]    PoisonIvy: Assessing Damage and Extracting Intelligence, FireEye, Inc. [white paper] 2013

[2]    Threat Report: Poison Ivy, Microsoft | Malware Protection Center. [white paper] 2011

[3]    Anatomy of an Attack, RSA Speaking of Security. [blog] 2011

Nest-ly Done

Honey-Do List, Item #2:  Install new thermostat  Install a new gadget that will change our lives in this house forever.Image

We are proud and, who some may portray as pretentious, 2nd Generation NEST Thermostat owners.  Depending on how successful you were with sharking a deal off an online retailer or auction, expect to shell out ~$200.  Are we crazy?  Shhh, hell yes, we’re crazy.  We’re crazy about saving some money and being green!  Ok, for those who know me, it’s more about saving money and convenience.  Green is a close 3rd and preferable by-product of many appliance purchasing decisions.  How much are we looking at saving?  We haven’t quite worked those numbers out, yet. However, I can tell you how convenient it is.  You see, our 3 story home on cement slab warms and cools significantly different based on the level you are sitting on.  To make a long story short, I am willing to pay for the convenience that allows my significant other to change the temperature from a phone, tablet, or computer without needing to ask me for help.  There are a ton of other features built into this fine piece of hardware for which I’ll cover another day.

My first impressions are; delightful packaging (imagine Apple getting into home heating and cooling appliances), easy to install, easy to configure, and easy to control.

Indicative of the digitized world we live in, any device with an embedded computing or operating system requires updates.  For the first 7-8 minutes the thermostat updated itself using my home internet connection.  Emblematic, no?

Signature here Sir…

Talking public key encyption here.

Does the order matter in which signature and confidentiality functions are applied to a message?  What order should they be applied and for what purpose?

In short, the answer is, Yes.

First, it is important to clarify the use of the term ‘signature’.  For this question it is assumed the signature would provide a message authentication function where Alice would encrypt a message (M) with the Alice’s private key (PRa).  Commonly, a hash of the message would be made and encrypted.  Upon receiving the message, Bob decrypts the message with Alice’s public key (PUa).  One of the fundamental constructs of public key encryption is that no one else except Alice has her private key.  Therefore, when Bob received the message encrypted with Alice’s PRa, Bob can safely assume it truly came from Alice.  Another fundamental of public key encryption is that everyone else including Alice has her public key.  The PUa allows Bob to decrypt any message from Alice encrypted with her private key.  In the same light, an unintended receipient like Evan the Eavesdropper can also decrypt Alice’s message to Bob.  When Alice encrypted her message to Bob with her PRa, she was solely providing integrity and non-repudation.  If Alice was concerned about confidentiality of the message, she would need to encrypt the message with Bob’s PUb so that Bob would be the only one who can decrypt the message using his PRb.

How do you decide which to first, sign and encrypt or encrypt and then sign?  To better explain the implication of order, we can explain what occurs when each is function is applied first.

Scenario 1. Authentication first, Secrecy second

1. Encrypt M with PRa to create Y

2. Encrypt Y with Bob the receiver’s public key (PUb) to create Z

What are the implications for the receiver in this scenario?

Bob will have confidence that the message Y is from Alice if the underlying construct of PKI is sound, meaning Alice’s private key PRa cannot be accessed by anyone but Alice.  Bob will have confidence that the message Z was kept secret from anyone who doesn’t have his private key PRb to decrypt the signed message.

Scenario 2. Secrecy first, Authentication second

1. Encrypt M with PUb to create Y

2. Encrypt Y with PRa to create Z

What are the implications for the receiver in this scenario?

Bob will have confidence that the message Y can only be decrypted by him assuming the underlying construct of PKI is sound, meaning Bob’s private key PRb cannot be accessed by anyone but Bob. Bob will NOT have confidence that Alice sent message Z without validating her digital signature (applying a message authentication function on message Y).  An adversary could intercept the message Z, strip off the signature and sign the same encrypted message Y and claim authorship.  Bob would be under the assumption the adversary (most likely disguising themself) knows the secret information and maybe less reluctant to share additional information.

Overall, what’s the better implementation?  It depends on the business case.  If message M needs to be encrypted at all times, then scenario 2 may be preferred.  A possible disadvantage to scenario 1 is that the public key algorithm would need to be computed 4 times, an expensive computation.


W. Stallings. 2011. Cryptography and Network Security 5th Edition. Upper Saddle River, NY: Prentice Hall.